Wednesday, October 14, 2009
J's and U's Updated / Speed Increases
I mentioned a few weeks ago that I planned on making I's/J's and U's/V's look the same on the back-end, while preserving their traditional orthographies on the front-end. I've just completed this task!
My main motivation for making this update is because certain passages stored in The Latin Library reflect the older conventions of using J's for consonantal I's or U's for both consonantal and vocalic V's. Numen's parsing engine was having trouble recognizing forms like jecit (iecit) and uuius (vivus). So now as a result -- after a bit of work -- the engine is updated and now recognizes more possibilities than ever. Incidentally, internally J's are stored as I's and U's are stored as V's.
Another project I completed at the same time is an order-of-magnitude speed improvement for parsing. I was trying to figure out ways to make the engine faster and I discovered a shortcut that boosts speed tremendously. When parsing a word, the engine used to spend between 250ms and 500ms parsing each word! That was always disappointing to me, but I had gotten around the problem by caching the results. Now, however, word parsing takes about 25ms!
Why bother improving the speed? Because soon I will be implementing word lists and frequency lists! A word list, of course, is just a "mini-lexicon" that defines only the words in your chosen passage, and a frequency list is a list of words in order of how often they appear in a passage. The word list will be helpful to quickly work on vocabulary for a passage, and a frequency list will help Latin students study more effectively by giving them the most frequent words first. I'm very excited about this feature, but I don't anticipate it will be done before January 10th (giving me the winter holiday to work on it).
That's all for now!
My main motivation for making this update is because certain passages stored in The Latin Library reflect the older conventions of using J's for consonantal I's or U's for both consonantal and vocalic V's. Numen's parsing engine was having trouble recognizing forms like jecit (iecit) and uuius (vivus). So now as a result -- after a bit of work -- the engine is updated and now recognizes more possibilities than ever. Incidentally, internally J's are stored as I's and U's are stored as V's.
Another project I completed at the same time is an order-of-magnitude speed improvement for parsing. I was trying to figure out ways to make the engine faster and I discovered a shortcut that boosts speed tremendously. When parsing a word, the engine used to spend between 250ms and 500ms parsing each word! That was always disappointing to me, but I had gotten around the problem by caching the results. Now, however, word parsing takes about 25ms!
Why bother improving the speed? Because soon I will be implementing word lists and frequency lists! A word list, of course, is just a "mini-lexicon" that defines only the words in your chosen passage, and a frequency list is a list of words in order of how often they appear in a passage. The word list will be helpful to quickly work on vocabulary for a passage, and a frequency list will help Latin students study more effectively by giving them the most frequent words first. I'm very excited about this feature, but I don't anticipate it will be done before January 10th (giving me the winter holiday to work on it).
That's all for now!
Labels: accuracy, database, development, features, frequency lists, google cache, orthography, parsing engine, performance, slowness, vergil, word lists
Wednesday, May 27, 2009
Speed Improvements
Sometimes I take a little bit of time off from reading and cogitating to work on important stuff -- stuff like speed improvements for this website.
This is incredibly nerdy stuff. It actually takes my mind off harder things. Don't ask!
The biggest improvements came in database queries. Some of the queries I was using were executing more slowly than I would have expected. In researching this problem I discovered something called prepared queries. I had no idea they would improve execution speed of certain queries by nearly 10x! On the back-end of things, that's a considerable improvement. On some pages it reduced the overall server load of each page by half -- to 35ms from 65ms! On the front-end, the site will probably feel a tiny bit snappier. Overall your average page load will reduce from about 160ms to about 130ms (since it takes about 100ms for intercommunicative data to traverse the internet from your computer to the server and back). That may not seem like much on your end (a 15% drop in latency) but on the server side it's quite dramatic (a 50% drop in latency).
This is incredibly nerdy stuff. It actually takes my mind off harder things. Don't ask!
The biggest improvements came in database queries. Some of the queries I was using were executing more slowly than I would have expected. In researching this problem I discovered something called prepared queries. I had no idea they would improve execution speed of certain queries by nearly 10x! On the back-end of things, that's a considerable improvement. On some pages it reduced the overall server load of each page by half -- to 35ms from 65ms! On the front-end, the site will probably feel a tiny bit snappier. Overall your average page load will reduce from about 160ms to about 130ms (since it takes about 100ms for intercommunicative data to traverse the internet from your computer to the server and back). That may not seem like much on your end (a 15% drop in latency) but on the server side it's quite dramatic (a 50% drop in latency).
Labels: ajax, database, development, prepared query, web server
Monday, February 23, 2009
Again with the optimizing! Oy!
I've been tinkering again.
First, I did some server-side wizarding and sped up the website by a small margin (maybe 25%) and also stopped a nasty bug that was slowing down pages every once in a while.
I fixed some smallish bugs on the search pages. The user interface should be a tad more useful and friendly there.
While designing a new, awesome feature (verb paradigms!) I uncovered some paradigm errors. Those have been neatly squashed.
Soon, you will see a masterpiece in action! Full paradigms for all parts of speech! Right now I've got the verbs mostly working. I'll unveil this feature when I feel it's good enough for prime-time.
That's all for now! Carry on...
First, I did some server-side wizarding and sped up the website by a small margin (maybe 25%) and also stopped a nasty bug that was slowing down pages every once in a while.
I fixed some smallish bugs on the search pages. The user interface should be a tad more useful and friendly there.
While designing a new, awesome feature (verb paradigms!) I uncovered some paradigm errors. Those have been neatly squashed.
Soon, you will see a masterpiece in action! Full paradigms for all parts of speech! Right now I've got the verbs mostly working. I'll unveil this feature when I feel it's good enough for prime-time.
That's all for now! Carry on...
Labels: bugs, development, paradigms, web server
Wednesday, December 24, 2008
New Server and Speed Increases
It's finally here and up and running!
I bought a new server. Did you know you can get slightly older computers, but still really powerful, for super cheap? People and businesses upgrade and then basically give their computers to discounters for nothing! I got this server for $144, with tax, shipping and an extra year's warranty. I'm very impressed.
Also, U.N.M (University of New Mexico) gave me a static IP address on their network, so we have a super-fast internet connection.
So, if you're used to this site being slow, get ready for serious changes! In general, moving to this new server on this new internet connection has increased the speed by an order of magnitude (from 300ms per request to 15ms per request). Wow!
But that's not all folks! I've also done some back-end coding to cache the results of morphology lookups. So now morphology lookups should increase by another order of magnitude (as long as a word is cached). If the word is not cached, the lookup will still be 2-3 times faster.
I apologize for geeking out a bit here, but I hope you notice the speed improvements.
As usual, I'm always developing The Latin Lexicon, but since I'm on winter break, expect to see some serious improvements for January!
Oh, one more thing. I also set up some bug-tracking software (BugZilla) to keep track of issues and improvements. So if you find all this technical stuff interesting, feel free to check it out!
Ok, one more thing! OpenID logins will be down for a day or two. Also, if you created an account or any flashcards between the 16th of December and today, I'm afraid that information is lost because I upgraded the database on the 16th and didn't get it moved until today. Sorry about that, if you're affected.
I bought a new server. Did you know you can get slightly older computers, but still really powerful, for super cheap? People and businesses upgrade and then basically give their computers to discounters for nothing! I got this server for $144, with tax, shipping and an extra year's warranty. I'm very impressed.
Also, U.N.M (University of New Mexico) gave me a static IP address on their network, so we have a super-fast internet connection.
So, if you're used to this site being slow, get ready for serious changes! In general, moving to this new server on this new internet connection has increased the speed by an order of magnitude (from 300ms per request to 15ms per request). Wow!
But that's not all folks! I've also done some back-end coding to cache the results of morphology lookups. So now morphology lookups should increase by another order of magnitude (as long as a word is cached). If the word is not cached, the lookup will still be 2-3 times faster.
I apologize for geeking out a bit here, but I hope you notice the speed improvements.
As usual, I'm always developing The Latin Lexicon, but since I'm on winter break, expect to see some serious improvements for January!
Oh, one more thing. I also set up some bug-tracking software (BugZilla) to keep track of issues and improvements. So if you find all this technical stuff interesting, feel free to check it out!
Ok, one more thing! OpenID logins will be down for a day or two. Also, if you created an account or any flashcards between the 16th of December and today, I'm afraid that information is lost because I upgraded the database on the 16th and didn't get it moved until today. Sorry about that, if you're affected.
Happy holidays! Happy Hannakwanzaamas!
Labels: bugs, development, features, flashcards, openid, slowness, UNM, web server
Monday, July 28, 2008
Into the Great Wide Open
There's still a lot of work to be done on this site, but the main dictionary features are in place. You can search in Latin and search in English, you can browse Latin words, and coolest of all is the word study tool. The word study tool allows you to type (or copy and paste from any Latin source such as The Latin Library) up to 50,000 words -- and see the words parsed, identified and defined.
Another cool feature, one which is in development, is the flashcards feature. Anytime you see a word you want to study, just check the "I want a flashcard" option. Then, you can print out a list of your flashcards on Avery Business Cards! A planned feature is to be able to study your flashcards online, and keep "sets" of flashcards.
One feature which is not ready yet -- but coming soon -- is the paradigm creator. When completed, The Latin Lexicon will create a full paradigm for any word in the dictionary!
Currently under development is The Latin Lexicon for iPhone/iPod touch. If you have one of these fantastic little devices, give it a try!
There's a lot to come, so this application will remain in beta for a while. Even so, I hope you find it useful!
Another cool feature, one which is in development, is the flashcards feature. Anytime you see a word you want to study, just check the "I want a flashcard" option. Then, you can print out a list of your flashcards on Avery Business Cards! A planned feature is to be able to study your flashcards online, and keep "sets" of flashcards.
One feature which is not ready yet -- but coming soon -- is the paradigm creator. When completed, The Latin Lexicon will create a full paradigm for any word in the dictionary!
Currently under development is The Latin Lexicon for iPhone/iPod touch. If you have one of these fantastic little devices, give it a try!
There's a lot to come, so this application will remain in beta for a while. Even so, I hope you find it useful!
Labels: browse, development, features, flashcards, iphone, ipod touch, paradigms, search