Today I implemented a simple log for the cache and a corresponding report in the "administration" panel for the website. As you can see the Study page caching, which was meant to improve the page loading time, hasn't been very effective so far:

That's because the cache only lasted an hour, and with visits spread across thousands of kanji pages, the cache typically wouldn't last long enough for another visitor to benefit from it.

I've now set the cache of the shared stories list to 12 hours, and the effectiveness is closer to 30% as of writing. Now I have real numbers, and hopefully we will see at least every second request to the Study page to benefit from the cache.

It's a good thing I finally looked into this because the queries for the Shared Stories list have become rather slow. I've seen response times as high as three seconds and typically from one to two seconds... which is actually really bad. I'm not at all an expert in database design... my hunch is that because the stories take several hundred megabytes.. the queries now have to span across a lot of storage and have become slower and slower over time.

The cache will buy me some time to think about better solutions. At some point I'm going to have to think about what to do with all those private stories. One solution to trim the database that I have considered, is to move private stories for very old and unused accounts to a second database. Since the "backup" database will be essentially dormant it should have no impact on performance. Then, if a user tries to sign in an "expired" account, they could export their data from the second database, and/or the account could be restored transparently without the user knowing anything.

The timezone setting for the server was set incorrectly, so that due cards and "reviewed today" count would not update at the usual time. This is now fixed. Remember to restore your timezone setting in the Account page if you tweaked it yesterday. Thanks for faneca and Daniel for the report!

Update : three minor fixes:

• Fixed broken /sightreading URL
• Fixed broken article link in Manage page to the RTK Supplement news update
• Fixed error when clearing the learned kanji list ("Clear" in Study sidebar)

I haven't got any other emails since the update so it looks like the update went smoothly!

The website has been updated today with changes that should improve the loading times of the Study pages.

The Shared Stories list on the study pages is now cached. I have set the "lifetime" of cached stories initially as low as one hour. So eg. if you share a new story, or upvote someone else's story, it may take up to an hour to be reflected in the page. Update: it may take several hours, up to a day to notice a difference because the cache is very conservative atm (probably too much) .

This is obviously a usability problem. How do you signal to users that the content is not immediately updated, but will be eventually? It's not ideal, but I think the tradeoff is worth it. It's not critical for votes, reports, and shared stories to be reflected in real time on the website, and this saves a lot of unnecessary queries on the database.

I will keep an eye on just how many pages are cached, based on the user activity, and then increase gradually the effect of the cache until it provides a significant speed improvement. In theory, updating the Shared Stories list even just once a day should provide more significant gains. But I wanted to think about how that is going to work usability wise so I will finetune it over time.

Since the website has been entirely upgraded to a more complete framework, there may be a few bugs that slipped through my tests. These should be very easy to fix. If you find an issue please let me know and I will fix it asap.

Last month I worked with member porukkusu on a simple API that lets a third party application connect with a Kanji Koohii user's account. The Android application, called Kanji Ryokusha is now available on slideme.org!

The app lets you complete SRS reviews on your mobile device. The key feature of the app is that you can write the kanji on your touch enabled device, before flipping the flashcard and comparing your answer with the correct one.

If you use the Kanji Koohii reviewing system and own an Android device, give it a go, and be sure to give some feedback to porrukusu in this thread!

A few points I need to mention:

I struggled implementing OAuth last month, partly because I was just so tired and I couldn't find any straightforward solution in the form of a library that handles OAuth completely from the server side. For this reason although there is a simple API working on the website for the SRS reviews, it is not made public yet. However if you have an interest in it, just get in touch as more developers can help me design it better.

Without OAuth we decided to go ahead with a simple sign in through the website's own sign in page. In programming terms I believe this is "good enough" for the time being. Hopefully in the future I can improve the API authentication, and at that point I will make it public. The downside to this is that the app needs your username and password to login.

In terms of privacy: the API currently only gives access to your flashcards (read/write) and custom keywords (read only). Your email is not shared with the app. If you decide to stop using the app you can always update your password if you'd like to be 100% sure it is no longer accessible through the third party app.

Wow.. I am done refactoring the website to the symfony framework! I'm actually surprised how easy it has been and wish I had done it sooner. I started September 3rd, and I have now converted all the pages, the build scripts, the deployment script. Everything seems to work fine. It took ~11 days and a little under 30 hours total, which is pretty impressive.

So this is an interesting experience. Years ago when I wanted to make my own "lightweight" MVC to learn more about php and web development... I knew refactoring was a big pain. And so I made a fortuitous decision to use a well designed API as a guide instead of trying to make my own from scratch. It was a great learning experience. I basically implemented all the basic MVC components from symfony such as Request, Response, Context, and so on. I left out everything I didn't need or I didn't understand. I also included classes from symfony as is where that made sense, for example the routing system, which didn't make sense for me to try to simplify. I was basically taking a grand tour through an MVC, and through symfony's code I was learning about all kinds of php functions and programming tricks.

Since I followed a well designed framework's conventions, the "actions" and "views" where much of the magic happens required practically no changes. I use a Zend_Db like API for the database, and a bunch of other custom classes so I had to create a bridge between my classes and those of symfony, but for the most part it's pretty clean and makes sense to me. Though the website is not entirely written in the symfony "style", there is no overhead. Either my classes, or symfony's classes are used depending on the type of functionality. For example I have a simpler version of the "sfView" class which I use for rendering many widgets such as the ajax tables.

So this is great. I now I have access to a much more complete framework, and in particular powerful caching capabilities. In the coming week I am going to explore those caching features and try to speed up the Study pages as much as I can.

One thing that often puts me off writing updates is that I'll just stare at the screen and draw a blank. Come to think of it, it is closely related with the absence of planning and clarity as to what I'm working on.

First, I planned some changes to the SRS described earlier. Then came surgery for my deviated septum (difficulty breathing through the nose) and lost track of my goal as my sleep quality has only slightly improved since. I now breathe more consistently but the air flow is only 50% of what I consider comfortable so I'm still breathing through the mouth and I never really know how I'll feel when I wake up. Heck I just had two days of decent sleep in a row which is almost a blessing. I'm told it can take months for this surgery to show its final results.

Someone got in touch about creating an Android app that connects to the site. Since I put off making an API for so long, I thought I'd finally get down to it. We have a very simple API working, and you may hear soon about an Android app that allows to do SRS reviews synced with your Kanji Koohii account. Unfortunately since I was struggling with my sleep and overall energy... I just couldn't wrap my mind around all the abstraction in OAuth and the API doesn't have a proper authentication for the consumer (the app, not the user). I couldn't come up with a simple alternative myself either. Currently the app takes your Kanji Koohii login credentials. It's not ideal since in theory the app author can store those credentials and mess with your account. But then again there is always some level of trust required when using an app that handles your data. Many apps do this (take your login and password), and all things considered.. nothing too nefarious will happen with a Kanji Koohii account. When you login, your email is not shared by the API. The API currently only deals with the SRS status and flashcards. So private stories aren't a concern either. However if I want to make the API available for general consumption, and expand it I will have to implement OAuth or something similar.

While working on this I also realized the website's MVC (Model-View-Controller) framework is really out of date. Incredibly.. the site has been running on a very trimmed down version of symfony 1.2 for the past ten years. Symfony is actually a great framework, but I have been using a hacked version of my own rather than the full framework. It was a great experience for learning php: I used the symfony API as a tutorial, sort of, and implemented all the core MVC components by copying much of the code, leaving out a lot of complexity. Thus, I was learning both php and MVC from the inside out. The reasoning was that at some later date, if I wanted to I could refactor the site with the full framework since the API is relatively similar. So now I am finally doing this.

This is the kind of updates that frustrating to work on because I have nothing to show. However I think it will be well worth it because I've noticed at certain times of day the website can become slow, and this change will finally bring some much needed caching. I think the Study section is the most expensive on the server side. It includes a handful of queries per page. Caching of course is complicated on a site like this since a lot of the presentation is dynamic content, meaning every user sees different things. I think the shared stories list is a great candidate for caching, since that list is more or less static for each kanji, and it's not critical for the the votes/reports to be updated in realtime. So I am excited to see how this will improve the responsiveness of the site.

Upgrading the framework also could help me with the API. I may be able to plug in OAuth support from an already made library for symfony.

Finally, I will have a proper internationalization and localization support with I18N. I have no idea what that thing is, but hey.. it means the likelyhood of translating the site to other languages will be much more feasible. It's just one of those things that I never added to my "trimmed down" version of the MVC framework.