I finally added a link to each individual post... (sigh). It's important for search engines and to let users share those posts. Just one of those things that sat at the bottom of a to do list somewhere. Anyway, I've rolled my eyes enough times already. :P Point is, it's there.. so if you wanted to share a link to one of the news posts, the title of this news post is now a link !

By the way, the background color changed recently. Did anyone notice? :D I thought it was nicer on the forum so I brought the change to the main site as well. In this old post you can see the difference... Hopefully you'll agree it is nicer. I think it's funny... it seems I remove a little bit of yellow from the background color every few years. It took ten years to get to this point!

PS: Apologies to our Brazilian friends! I've noticed once in a while people have trouble answering the question on the sign up page. They are often Portuguese spellings (and probably misspellings too) for the name of the capital of Japan. I have just added more of those spellings to make the registration easier.

I've started work on making the website responsive and more usable on mobile devices.

A quick glance at Google Analytics shows that a significant 24 % of visits are from "mobile", and another 8% from "tablet". I guess a "tablet" could be considered a small desktop PC and usually most websites will fit into a tablet display without a "mobile" view. As for the "mobile" devices, the top 50% of those are shared between the iPhone (31%) and iPad (17%). The other 50% are spread thin between hundreds of different devices and models such as Samsung Galaxy, Google Nexus, Kindle; Motorola, Huawei, etc.

As far as I can tell, the website works fine on tablets. When I redesigned the main navigation in early 2015, I purposely made it to work with touch. That way the drop down menus can be used on a tablet where there is no "mouse over" functionality.

Much of the work consists of allowing the content to "collapse" into a single column. The Study page is a good example. I suppose a good design would be for a search box to appear at the top, followed by the user's editing area, and then the shared stories.

As a single developer, I don't want to manage two entirely different websites. I can't afford to keep in sync a "mobile" optimized website, and a desktop website. So what I am trying to do, is a responsive website, that will be more comfortable to use on mobile devices (typically portrait view on a handheld phone). That means from the server point of view, there is only one page. The stylesheet is used to fit the same page content to different displays. In order to achieve that, I need to find a good compromise between both.

One tricky problem to solve are the tables. What should I do with them? We can't fit all the information from the Flashcard List in portrait view on a phone. Should I just hide some of the less important columns? Do mobile phone users expect that the mobile site will feature a "stripped down" version of the information? I really don't know. What if a visitor has never visited the site other than on their mobile device? Do they know that the tables would show more information when seen on a desktop browser or tablet (large display)?

Fortunately the Flashcard Review page is already optimized for mobile devices!

Some pages such as "Custom Review" are already responsive. The content can simply collapse into a single column. It's easy and fast to scroll on touch devices so that works out well.

Other pages like "Manage" featuring a sidebar navigation currently also collapse the sidebar. However the whole sidebar navigation shows at the top. I suppose here the main improvement would be to add a button that expands /collapse the navigation, while opening the most common sub page by default. This way the main content is not pushed too far down, and is seen when the page loads.

Do you have any suggestions for the mobile version of the site? Let me know!

Results of the performance update. I think the last performance update was a success! I'm definitely seeing a difference: most Study pages now return in less than half a second and that includes the full return trip to the server. When a Study page is not cached, the queries are still a bit slow to my liking, but still better than before so I'm satisfied with the results. If it becomes necessary, I figured another way I could improve the speed further but this update bought me a lot of time that I can hopefully focus on other things.

I fixed a rather big bug today related to the On / Kun example readings. The reading was cut after the highlighted part... The On / Kun reading corresponding tio the kanji flashcard was correct, but the full word was not spelled out so the reading could be misleading. For example 少 ("few") would show the reading すこ ... however that reading only made sense in the full word すこし. Now the full word is displayed correctly, along with the highlighted kanji reading. Thanks to Noah for letting me know!

To enable example words on your flashcards go to Account Settings > Flashcards. Example words are selected based on their frequency of use as documented in JMDICT (roughly 16000 entries in JMDICT). For common kanji where there are several suitable On and/or Kun readings, the example words will be picked at random. That way you can get more of your kanji reviews after completing RTK.

Today's maintenance update went well... but not without hurdles. The results are not as significant as I hoped to be perfectly honest, but overall the return time of Study pages is definitely better.

Before the database update I routinely saw queries on the Study pages that took 1.5 seconds, so adding the 300 ms of response time a page would be returned in two seconds, sometimes more! (300 ms is an average for me in Belgium, with the servers located in Texas; USA).

After the update it looks like most pages return within one second. The queries unfortunately can still vary wildly but overall it seems we have at least a 50% improvement in response time.

Read on for the gory details!

In addition to that the caching of shared stories has been improved. Now it truly saves the most expensive queries, and the cache has been made to last much longer. Since we invalidate (ie. refresh) the cached page whenever a public story is edited or voted, I made the cache to last for a month (as of writing). There are ups and downs to that. Study pages corresponding to the beginning of the RTK index are likely to see more activity with shared stories and votes, and hence the cache will not be as efficient for them. Time will tell !

Keep in mind that for a few days, most pages will still take the extra time to be generated because they are not cached yet. However as more users browse the pages, and more pages are cached you should see an improvement.

The main improvement likely comes from the redesign of the indexes, and getting rid of the "temporary table" problem. And who knows, it may be that on peak times the changes I made are even more beneficial since that would be when bottlenecks happen.

I'm sorry the update took so long (about 3 hours). As soon as I took the site down, I found out that the MySQL server does not allow partitions! (unlike my developmental version). This was a further performance improvement I had to undo in the code and database schema.

Some numbers: there are 361871 shared stories as of writing. There are 6209558 stories stored altogether, equivalent to a 1.1 GB MySQL table.

There are still other interesting ideas I'm brainstorming. One approach for example, to make up for the lack of partitions, would be to move the public stories into a new table. Whenever a story is made public it could be copied there, and that means MySQL would deal with a table that would be less than 100 MB instead of 1.1 GB, with all the public stories addressed by a SQL query clustered together.

I'm almost ready to make a big maintenance and performance update to the database. The goal is to improve the responsiveness of the Study pages which have gotten slower over the years since we are now storing 6.2 million stories from users all over the world!

The maintenance update and downtime MAY happen tomorrow Friday, early afternoon and/or evening CET time.

The update may take an hour or more. I need to download the full database backup before the changes just to be extra safe. Most of the changes are manual updates I do over SSH. Then, while I am the only person able to access the site, I need to go through a bunch of pages, add a story, vote a story, etc... to make sure there are no glaring bugs.