ETL
- Thread starter Leroy
- Start date
- Status
Leroy
2012 Troll of the Year
[quote1256075780=Keeripes!]
One phase? Entire process? How was it implemented? Was it a parallel algorithm? YOU LEAVE TOO MANY UNANSWERED QUESTIONS!!!!
[/quote1256075780]
One phase, multiple processes in a parallel algorithm. Bulk binded into memory, however, this might not fit higher row numbers, so we'll have to go to some sort of file-based stage.
My long term goal is 6 million rows in a second, but I think I will only realize this at the production level (we use ram disks for caching at prod level -- UBER FAST)
One phase? Entire process? How was it implemented? Was it a parallel algorithm? YOU LEAVE TOO MANY UNANSWERED QUESTIONS!!!!
[/quote1256075780]
One phase, multiple processes in a parallel algorithm. Bulk binded into memory, however, this might not fit higher row numbers, so we'll have to go to some sort of file-based stage.
My long term goal is 6 million rows in a second, but I think I will only realize this at the production level (we use ram disks for caching at prod level -- UBER FAST)
Are you developing an algorithm for a certain hardware configuration, or adapting/creating algorithm to streamline your current config or a general software package?
I lack the linchpin, the crux, my good sir... I lack context.
Once again, you introduce to me a new concept which for some reason I decide to go research instead of studying for my midterm du jour. End up with more questions than I start with.
If you are developing some sort of hot new code, please send all documentation to me
K. Herpez
1337 Noob Avenue
TB
Canada
7I7 5UK
so I may:
1) Learn Leroy's Secrets
2) Whiteout and replace his name on the patent paper
3) ???????
4) $$$PROFIT$$$$
I lack the linchpin, the crux, my good sir... I lack context.
Once again, you introduce to me a new concept which for some reason I decide to go research instead of studying for my midterm du jour. End up with more questions than I start with.
If you are developing some sort of hot new code, please send all documentation to me
K. Herpez
1337 Noob Avenue
TB
Canada
7I7 5UK
so I may:
1) Learn Leroy's Secrets
2) Whiteout and replace his name on the patent paper
3) ???????
4) $$$PROFIT$$$$
Steve
TD Admin | Bacon
*smacks self with shovel*
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
[quote1256125771=Steve]
*smacks self with shovel*
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
[/quote1256125771]
iloveustve
*smacks self with shovel*
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
[/quote1256125771]
iloveustve
dead mike
TD Member, Legend, Puncher of Faces, Chatbox King
[quote1256128827=Steve]
*smacks self with shovel*
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
[/quote1256128827]
Given the existence as uttered forth in the public works of Puncher and Wattmann of a personal God quaquaquaqua with white beard quaquaquaqua outside time without extension who from the heights of divine apathia divine athambia divine aphasia loves us dearly with some exceptions for reasons unknown but time will tell and suffers like the divine Miranda with those who for reasons unknown but time will tell are plunged in torment plunged in fire whose fire flames if that continues and who can doubt it will fire the firmament that is to say blast hell to heaven so blue still and calm so calm with a calm which even though intermittent is better than nothing but not so fast and considering what is more that as a result of the labors left unfinished crowned by the Acacacacademy of Anthropopopometry of Essy-in-Possy of Testew and Cunard it is established beyond all doubt all other doubt than that which clings to the labors of men that as a result of the labors unfinished of Testew and Cunnard it is established as hereinafter but not so fast for reasons unknown that as a result of the public works of Puncher and Wattmann it is established beyond all doubt that in view of the labors of Fartov and Belcher left unfinished for reasons unknown of Testew and Cunard left unfinished it is established what many deny that man in Possy of Testew and Cunard that man in Essy that man in short that man in brief in spite of the strides of alimentation and defecation wastes and pines wastes and pines and concurrently simultaneously what is more for reasons unknown in spite of the strides of physical culture the practice of sports such as tennis football running cycling swimming flying floating riding gliding conating camogie skating tennis of all kinds dying flying sports of all sorts autumn summer winter winter tennis of all kinds hockey of all sorts penicillin and succedanea in a word I resume flying gliding golf over nine and eighteen holes tennis of all sorts in a word for reasons unknown in Feckham Peckham Fulham Clapham namely concurrently simultaneously what is more for reasons unknown but time will tell fades away I resume Fulham Clapham in a word the dead loss per head since the death of Bishop Berkeley being to the tune of one inch four ounce per head approximately by and large more or less to the nearest decimal good measure round figures stark naked in the stockinged feet in Connemara in a word for reasons unknown no matter what matter the facts are there and considering what is more much more grave that in the light of the labors lost of Steinweg and Peterman it appears what is more much more grave that in the light the light the light of the labors lost of Steinweg and Peterman that in the plains in the mountains by the seas by the rivers running water running fire the air is the same and then the earth namely the air and then the earth in the great cold the great dark the air and the earth abode of stones in the great cold alas alas in the year of their Lord six hundred and something the air the earth the sea the earth abode of stones in the great deeps the great cold on sea on land and in the air I resume for reasons unknown in spite of the tennis the facts are there but time will tell I resume alas alas on on in short in fine on on abode of stones who can doubt it I resume but not so fast I resume the skull fading fading fading and concurrently simultaneously what is more for reasons unknown in spite of the tennis on on the beard the flames the tears the stones so blue so calm alas alas on on the skull the skull the skull the skull in Connemara in spite of the tennis the labors abandoned left unfinished graver still abode of stones in a word I resume alas alas abandoned unfinished the skull the skull in Connemara in spite of the tennis the skull alas the stones Cunard
*smacks self with shovel*
Fuck I hate writing children lit essays so much its such a waste of time instead i think i will rant on the td forums to annoy all the grammar nazi faggots who view the forum not to mention i just made a awesome sandwich and smoked a bowl i cant wait to get done with school in the next year or i sure hope the global economy gets better before i get out i totally just got done owning lowbudget in team fortress two i stabbed him like twenty times he was so raged i loled so hard i almost forgot to mention that brains is a huge fag and i hate spinny man spinny is such a fag the only thing fagger then spinny is two men eating cherrys off each others assholes man i really dont want to write this stupid essay what a waste of my time and omfg this stupid infomercial wont end ngiht time tv is so lame did anyone really read all of this?
[/quote1256128827]
Given the existence as uttered forth in the public works of Puncher and Wattmann of a personal God quaquaquaqua with white beard quaquaquaqua outside time without extension who from the heights of divine apathia divine athambia divine aphasia loves us dearly with some exceptions for reasons unknown but time will tell and suffers like the divine Miranda with those who for reasons unknown but time will tell are plunged in torment plunged in fire whose fire flames if that continues and who can doubt it will fire the firmament that is to say blast hell to heaven so blue still and calm so calm with a calm which even though intermittent is better than nothing but not so fast and considering what is more that as a result of the labors left unfinished crowned by the Acacacacademy of Anthropopopometry of Essy-in-Possy of Testew and Cunard it is established beyond all doubt all other doubt than that which clings to the labors of men that as a result of the labors unfinished of Testew and Cunnard it is established as hereinafter but not so fast for reasons unknown that as a result of the public works of Puncher and Wattmann it is established beyond all doubt that in view of the labors of Fartov and Belcher left unfinished for reasons unknown of Testew and Cunard left unfinished it is established what many deny that man in Possy of Testew and Cunard that man in Essy that man in short that man in brief in spite of the strides of alimentation and defecation wastes and pines wastes and pines and concurrently simultaneously what is more for reasons unknown in spite of the strides of physical culture the practice of sports such as tennis football running cycling swimming flying floating riding gliding conating camogie skating tennis of all kinds dying flying sports of all sorts autumn summer winter winter tennis of all kinds hockey of all sorts penicillin and succedanea in a word I resume flying gliding golf over nine and eighteen holes tennis of all sorts in a word for reasons unknown in Feckham Peckham Fulham Clapham namely concurrently simultaneously what is more for reasons unknown but time will tell fades away I resume Fulham Clapham in a word the dead loss per head since the death of Bishop Berkeley being to the tune of one inch four ounce per head approximately by and large more or less to the nearest decimal good measure round figures stark naked in the stockinged feet in Connemara in a word for reasons unknown no matter what matter the facts are there and considering what is more much more grave that in the light of the labors lost of Steinweg and Peterman it appears what is more much more grave that in the light the light the light of the labors lost of Steinweg and Peterman that in the plains in the mountains by the seas by the rivers running water running fire the air is the same and then the earth namely the air and then the earth in the great cold the great dark the air and the earth abode of stones in the great cold alas alas in the year of their Lord six hundred and something the air the earth the sea the earth abode of stones in the great deeps the great cold on sea on land and in the air I resume for reasons unknown in spite of the tennis the facts are there but time will tell I resume alas alas on on in short in fine on on abode of stones who can doubt it I resume but not so fast I resume the skull fading fading fading and concurrently simultaneously what is more for reasons unknown in spite of the tennis on on the beard the flames the tears the stones so blue so calm alas alas on on the skull the skull the skull the skull in Connemara in spite of the tennis the labors abandoned left unfinished graver still abode of stones in a word I resume alas alas abandoned unfinished the skull the skull in Connemara in spite of the tennis the skull alas the stones Cunard
EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS EPIC POST IS ..my god I've gone cross-eyed..
Fork Included
TD Admin
i'm having tea...
Leroy
2012 Troll of the Year
[quote1256139636=Keeripes!]
Are you developing an algorithm for a certain hardware configuration, or adapting/creating algorithm to streamline your current config or a general software package?
I lack the linchpin, the crux, my good sir... I lack context.
Once again, you introduce to me a new concept which for some reason I decide to go research instead of studying for my midterm du jour. End up with more questions than I start with.
If you are developing some sort of hot new code, please send all documentation to me
K. Herpez
1337 Noob Avenue
TB
Canada
7I7 5UK
so I may:
1) Learn Leroy's Secrets
2) Whiteout and replace his name on the patent paper
3) ???????
4) $$$PROFIT$$$$
[/quote1256139636]
Context:
OLTP -> ODS -> OLAP
We are updating our ODS every 15 minutes from our OLTP system. We have a functional requirement to return claims information in under 2 secs, so we need a light method of copying data from our OLTP to ODS. We use Oracle's CDC to capture the changes and provide a view. From there, I come in, pull these changes, and push them into our ODS.
I designed the "algorithm" from the standpoint, of how to pull/push the data, and coded it (yay plsql), optimized the ODS system for the data, and whammo, was getting reported times. Its actually not a lot of data that we're moving (besides row count), however, we have to maintian referential integrity, and most importantly security (the data must not leave our Oracle)
Anyways, pseudo code documentation as follows.
1. Disable target table constraints
2. Disable target table indices
3. bulk collect all data for a given range from the source CDC tables (that is, the data that has changed), and store in a memory structure.
4. "push" (Best way to describe it, really), all in one go
5. Enable table constraints
6. Rebuild indices
So, this is how your average DBA with a coding background would do it.
However, I am not your average DBA, and my coding background is... ahem.. extensive
So I started pulling out some crazy times, by multi-threading the data push mechanism. I fired up C# (Only because it was on my computer, and didn't really want to jump through teh hoops to make C++ work nicely with Oracle) and each thread sends the range of data in its own process. Also, the session is not shared between threads. Playing around with thread counts, I found it performed best with 3-4 threads, although I came close with around 10 threads. I think this is just the nature of the dev system, and the limitations of the Oracle latching mechanism.
Its so enjoyable, I think I might whore myself out on a couple contracts for data movement.
Are you developing an algorithm for a certain hardware configuration, or adapting/creating algorithm to streamline your current config or a general software package?
I lack the linchpin, the crux, my good sir... I lack context.
Once again, you introduce to me a new concept which for some reason I decide to go research instead of studying for my midterm du jour. End up with more questions than I start with.
If you are developing some sort of hot new code, please send all documentation to me
K. Herpez
1337 Noob Avenue
TB
Canada
7I7 5UK
so I may:
1) Learn Leroy's Secrets
2) Whiteout and replace his name on the patent paper
3) ???????
4) $$$PROFIT$$$$
[/quote1256139636]
Context:
OLTP -> ODS -> OLAP
We are updating our ODS every 15 minutes from our OLTP system. We have a functional requirement to return claims information in under 2 secs, so we need a light method of copying data from our OLTP to ODS. We use Oracle's CDC to capture the changes and provide a view. From there, I come in, pull these changes, and push them into our ODS.
I designed the "algorithm" from the standpoint, of how to pull/push the data, and coded it (yay plsql), optimized the ODS system for the data, and whammo, was getting reported times. Its actually not a lot of data that we're moving (besides row count), however, we have to maintian referential integrity, and most importantly security (the data must not leave our Oracle)
Anyways, pseudo code documentation as follows.
1. Disable target table constraints
2. Disable target table indices
3. bulk collect all data for a given range from the source CDC tables (that is, the data that has changed), and store in a memory structure.
4. "push" (Best way to describe it, really), all in one go
5. Enable table constraints
6. Rebuild indices
So, this is how your average DBA with a coding background would do it.
However, I am not your average DBA, and my coding background is... ahem.. extensive
So I started pulling out some crazy times, by multi-threading the data push mechanism. I fired up C# (Only because it was on my computer, and didn't really want to jump through teh hoops to make C++ work nicely with Oracle) and each thread sends the range of data in its own process. Also, the session is not shared between threads. Playing around with thread counts, I found it performed best with 3-4 threads, although I came close with around 10 threads. I think this is just the nature of the dev system, and the limitations of the Oracle latching mechanism.
Its so enjoyable, I think I might whore myself out on a couple contracts for data movement.
- Status