KDB - Historical Table - what's the “right” way to doing it?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP


KDB - Historical Table - what's the “right” way to doing it?



I have a point of sale system that exports my data to a .csv which I then import into KDB. Currently what I do is export all the data from the POS to the csv and then create a single table. I have about 10 months of sales data and my csv file is about 11mb. As time grows I imagine that the csv file will be really huge and I wonder if this is inefficient.



At my old job what we would do is have a table for each individual day of data and then there would be an _hist table which would combine all the daily files. So if I wanted to just look at the data for the day I would look at the invoicedata table and if I wanted to look across all time I would look at the invoidata_hist table and set the query to look at date within (dateA;dateB). I'm wondering if I should set things up this way rather than the way I'm doing it now.



Am I better off having one very large csv file with all the data in it or should I create a csv file for each individual day? If the second way is better can anybody give me an idea of the best way to set this all up?



Thanks!




1 Answer
1



If your total number of records are not going to exceed million then splaying/partitioning might be an overkill.



Having said that there are multiple ways you can store the data on disk apart from storing data as CSV, check out this link



Since you asked for a date partitioned in the question, actually there are different ways you can partition your data :



You might want to store the data in a monthly partition based on the table count.



For saving the data to a partition, you can use .Q.dpft function


.Q.dpft[directory;partition;`p#field;tablename]



example for code.kx:


q)trade:(sym:10?`a`b`c;time:.z.T+10*til 10;price:50f+10?50f;size:100*1+10?10)
q).Q.dpft[`:db;2007.07.23;`sym;`trade]
`trade






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Arduino Mega cannot recieve any sketches, stk500_recv() programmer is not responding

Image
Clash Royale CLAN TAG #URR8PPP Arduino Mega cannot recieve any sketches, stk500_recv() programmer is not responding Yesterday we were trying to update the firmware of our 3D-printer a Tevo Tarantula i3 The printer was running Marlin 3D Printer Firmware. We installed a new module a TL-Touch Sensor, a chinese copy of the Bl-Touch an auto bedleveling touch sensor for a 3D printer. After re-configurating a new version of the firmware for our setup in the Arduino IDE we went on with uploading the sketch to the Arduino Mega that was controling our 3D printer. Before we started uploading we made sure the right COM-port was selected in the Arduino IDE and that also the right type of Arduino was selected. The Arduino Mega 2260. Then we started with compiling the firmware and then uploading it to the Mega, Somewhere in between that moment the IDE freezed and crashed. As a response we removed the USB cable that was connected from the PC uploading the firmware onto the Arduino Mega, (This was a st...
Read more

Visual Studio Code: How to configure includePath for better IntelliSense results

Image
Clash Royale CLAN TAG #URR8PPP Visual Studio Code: How to configure includePath for better IntelliSense results so I am a complete beginner to using Visual Studio Code and I have no clue what I am doing. I've searched around (maybe not enough), but I cant find just a simple explanation for someone like me on how to configure the c_cpp_properties.json file that I am redirected to whenever I click on the yellow lightbulb next to a line that is underlined with a green squiggle. Lightbulb example c_cpp_properties.json I just want to know what to put in the .json to make IntelliSense work properly This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. 2 Answers 2 From: https://code.visualstudio.com/docs/languages/cpp Below you can see that the MinGW C++ include path has been added to browse.path for Windows: { "name": "Win32...
Read more

C++ virtual function: Base class function is called instead of derived

Image
Clash Royale CLAN TAG #URR8PPP C++ virtual function: Base class function is called instead of derived I have following code snippet: using namespace std; class base { public: virtual void print(char a){ std::cout<<" Base "<<std::endl;} }; class derived :public base { public: void print(floata) {std::cout<<" Derived "<<std::endl;} }; int main() { base* d = new derived; d->print(1.5); return 0; } Output is "base" Why output is coming from base function and not from derived one By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
Read more