After the recent acquisition by CrowdGather, we are in the process of moving our entire network to their datacenter in Los Angeles. CrowdGather made a significant investment into new hardware to support freeforums.org and position us for future growth. Our engineers have been spending 10 hour days the past month preparing the new network and starting the migration. As of Today, we have moved approximately 30% of our forums to the new network. We will continue working around the clock to give you fast, reliable forum hosting. Now for some pictures:

Rack holding the new freeforums.org network

Network router and switches

Database & web servers

Storage area network (SAN) for storing files

Another view of database & web servers

We will continue to keep everyone in the loop regarding the network upgrade. Keep checking our announcement board for updates. Thank you for choosing FreeForums.org!

We are constantly being asked what we are working on next. Ever since we launched back in February of 2007 we have been at the top in terms of innovation and features. I would like to take a few minutes to discuess what we have in store for you, our customer:

Premium Service — In the coming weeks we will give you an option to purchase a “Premium Forum” for a small monthly cost. This will include additional features and mods not possible on the free system.
Simple Admin Panel — Lets face it, the phpBB3 admin panel wasn’t designed for non-tech savvy users. In the next two months we will be creating a simplified version of the admin panel that you can choose to use.
Make Money! — You put a lot of work into your forum because, like us, you are passionate about what you do. But who would complain about a little extra spending money for your time? Over this coming summer we will launch various ways of making money off your forum.
More Mods! — Can you ever have enough mods? We don’t think so either. With our new development cycle we’ve been releasing new mods in record time.

As always we will be working hard to bring you the very best tools and resources to run your community. If you would like to see anything added please let us know!

Since July 21, focus shifted at FreeForums in the way we thought and went about seeing our future. We wanted to make sure that what happened that night could never happen again. At the same time, we were seeing problems brewing in the air.

Database Clusters
Due to the rate at which FreeForums was growing and the limitations of MySQL replication, we saw ourselves running into a very severe road block. To maintain our MySQL servers, at the time we had one master MySQL server and several slave servers. All writes would go to the MySQL master which would then be synchronized to the MySQL slaves which all reads went to.

The problem we saw was that MySQL replication, on the slaves, was only capable of using a single CPU core to process updates coming from the master. To handle the operation of replication, the master server needed use of all its cores, yet each slave could only use one. Eventually we would hit a wall at 100% usage of the core and thus be unable to process any more updates. The result from this: slaves would fall out of sync.

Our solution to this was to break FreeForums up into multiple independent MySQL clusters. This would distribute writes over serveral master servers and allow us to add even more slave servers. But writing a system for this would not be easy and would take some extensive testing and development.

All the while we were developing this new system, we began hitting the writing bottleneck on our slaves. Throughout December, we were having slaves fall out of sync whenever there was high traffic hitting our web servers; however, thanks to the holidays, we were given two weeks of lower traffic which gave us more time to test and develop the system. Come January, however, that went away and for the entire first week and a half, slaves were out of sync every day from 11AM thru 7PM.

Given the obviousness that we were out of time, we halted testing of the second cluster system and launched it. The tests had been going well, and we wanted to continue testing a few more things to make certain that the system would work as expected, but we were out of time. On the morning of January 15, while watching Cluster-0 slaves fall out of sync at 6AM, I decided that we had to launch the system immediately. At that time, we picked 39 databases accounting for 20% of our daily traffic and sent them to the new cluster.

Low and behold, the Cluster0 slaves synced back up, and since that date we haven’t had a single slave fall out of sync. At this time, we have finished testing and development of the clustering system and will be using it from now on. The addition of clustering was a huge step forward for us as it will guarantee the speed and stability of FreeForums in the coming years.

Web Servers
The second problem we were already facing was that we needed a more stable environment to serve the websites that we were hosting. We had multiple web servers, but the way in which we were load balancing them was by the use of round robin. Round robin is a method where in your DNS records, used by the Internet to look up where your website is located, you list multiple IP addresses. When you type in your web address, your computer randomly selects from that pool of IP addresses and that is the server you land on. Problem here is that there is no load distribution and there is no failover protection.

Those two issues would spell chaos as FreeForums grew. Even though we had three web servers, one of them was receiving 50% of all traffic with the other two receiving 25% each. Next, what would happen if, say, the web server receiving 50% of traffic would go down? The answer is simple: 50% of viewers wouldn’t be able to access the forums they were trying to view.

The solution to this was simple, in theory; however, it would take time to test and develop. Consulting with SoftLayer network engineers, we were directed to look at the Zeus ZXTM Load Balancer. Zeus ZXTM is a software suit that distributes requests for various TCP and UDP services based on specifications that you set for it.

What Zeus ZXTM would give us is a level of stability and redundancy on our web servers that has never been seen before my FreeForums.org and its users. By sending all requests to the Zeus ZXTM load balancing server, we are able to evenly distribute all requests over all web servers as well as redirect requests when a web server goes offline. This will make it impossible for a failed web server to ever be noticed by our viewers again.

As of January 26, all viewers are accessing FreeForums.org by way of the Zeus ZXTM load balancer, and we couldn’t be happier.

The Future
FreeForums.org is well on its way to making all points of service HA (high availability) by making every single service redundant with backup servers, but we still have a ways to go before we achieve this.

First, the global MySQL servers used to house the data shared between all clusters, while having a slave keeping mirrored copy of the data, is still setup in a Master+Slave method. Should the master go down, a human will need to convert the slave to a master server. The solution to this is to utilize MySQL Cluster, which allows for all servers in the mix to be masters. Once this is done, if the primary server ever goes down the others can take over immediately without anyone needing to intervene.

Next, the file storage server is currently standalone. There is no failover, there is no mirror. Just backups being made every fifteen minutes. But, aha!, yet another easy fix. We will be looking into using the Lustre File System as a means to create several file servers which act as one unified server. Should a server ever fail, go offline, or do anything that makes it unavailable, all requests to that server will be directed to the others. No one would ever know the difference.

Lustre is based off Sun Microsystem’s ZFS, a very powerful and robust filesystem. Lustre is currently in use by several of the world’s largest supercomputers, hosting hundreds of millions of files totaling petabytes in total space. So we’re confident that it can handle our humble hundreds of thousands of files.

Many more changes are coming in the future, and things are sure to be very exciting as we move into the future.

When we launched our Ad-Sharing update which made sponsors default on all forums, we promised the revenue would “allow us to improve on the already high level of services”. We’re pleased to say that we’ve done just that:

We’ve run into a few problems over the past few months with web servers going down here and there. This was caused by an uneven distribution in load caused by using Round robin to balance the traffic. Round robin randomly selects from a pool of IP addresses and that is the server you land on. Problem here is that there is no load distribution and there is no failover protection.

The solution to this was simple, in theory; however, it would take time to test and develop and a lot of money to purchase the equipment. After consulting various network engineers, we looked into the Zeus ZXTM Load Balancer. Zeus ZXTM is a software suit that distributes requests for various TCP and UDP services based on specifications that you set for it.

What Zeus ZXTM would give us is a level of stability and redundancy on our web servers that has never been seen before by FreeForums.org and its users. By sending all requests to the Zeus ZXTM load balancing server, we are able to evenly distribute all requests over all web servers as well as redirect requests when a web server goes offline. This will make it impossible for a failed web server to ever be noticed by our viewers.

We made the final purchase this past Tuesday and all viewers are accessing FreeForums.org by way of the Zeus ZXTM load balancer. We couldn’t be happier

Before I begin, I should note that this is a very technical and in-depth post. Good luck.

FreeForums.org launched in February of 2007, and has grown at a rapid pace ever since.  With that comes growing pains.

For the first many months, a very simple set was required to keep the forums moving along. Within a few months, the need for more power came, but still nothing a single Intel Xeon couldn’t handle. But that was bound to change. At the time that I came to FreeForums.org, the setup was as follows:

Web Server
Single Intel Xeon L5310 (quad-core, 1.60Ghz, 8MB L2 cache)
2GB DDR2-667 FBDIMM RAM
250GB SATA2 Hard Drive
Overall CPU Usage: 30%

Database Server
Quad AMD Opteron 8212HE (dual-core, 2.00Ghz, 1MB L2 cache)
16GB DDR2-667 FBDIMM RAM
250GB SATA2 Hard Drive
Overall CPU Usage: 20%

At the time there were several major flaws in design that originally did not pose a problem.

Non-RAID Data Storage
In the world of servers and corporations, data redundancy and protection is an absolute must. What RAID does is it takes multiple hard drives and configures them in such a way that the loss of a numbre of hard drives (due to failure, for example) will not result in data loss (this is true for all levels of RAID except for RAID0).

The importance of RAID became very obvious the night I began consulting for FreeForums.org. That night, I was called in because all attempts to syncronzie the new MySQL server with the existing server’s data failed. The InnoDB database (see below) was unable to be read by the new server no matter what the staff tried to do.

My solution was that if we couldn’t move the data file, then let’s export the data ourselves and send it over row-by-row to the new server. But there laid yet another problem: All of the tables were in a single database. MySQL’s SQL export utility “mysqldump” was unable to dump SQL data as fast as we needed it to (45 minutes per forum at 44,000 forums or so, total time: 3.77 years). So my solution was to write a script that would read the data from MySQL, create the proper schema for it, and send it to the new server.

On the night of July 21, while the script was running and moving forums (it had moved roughly 4000 forums), the old MySQL server crashed and would never come back up. The single hard drive housing the InnoDB data file (containing all the forum data) corrupted and failed. After trying for 14 hours to recover the data, it became obvious that the data was lost.

The solution to prevent this in the future was simple: utilize RAID. The first hardware change I made with FreeForums.org was to institute the usage of RAID10 on all MySQL servers. RAID10 is a level of RAID which utilizes the speed benefits offered by RAID0 but provides data protection at the same time. In RAID0, each hard drive is setup in a linear segment in which data is sent in blocks across all the hard drives. It allows you a very fast virtual hard drive, but it offers no data protection. Four 80GB hard drives setup in RAID0 will offer you 320GB of storage space, but if even one hard drive fails you will lose all your data. RAID10 allows you to gain the benefits of RAID0, but it gives you data protection by pairing up each segment with a RAID1 mirror. In this case, four 80GB hard drives will offer you 160GB of storage space, but if you lose a drive in any segment, you’re safe. You can lose one drive in every segment without data loss, but should two drives in the same segment fail, then you lose all of your data. But the chances of that happening are slim to none without you having enough time to fix it.

Another benefit of RAID is that it gets you added disk bandwidth. The more drives you have, the more bandwidth is available to you. For us, that is one definite plus.

Thanks to RAID, we will never have to worry about what happened on July 21 from ever happening again.

InnoDB Storage Engine in MySQL
InnoDB is a great storage engine when you have very large and active tables, but when you have an insane amount of tiny tables, InnoDB becomes a huge problem. Mainly, InnoDB uses a lot of system resources to read data from it’s main file (ibdata1). Back in July, we needed, at an absolute minimum, 16GB of RAM on the MySQL server to stay online for even 8 hours. But the truth of the matter was, we needed 32GB to remain stable, and that number would only grow as FreeForums.org grew.

The solution to this was a simple yet time-consumng task. Convert from the InnoDB storage engine and to MyISAM, which is more suited to our needs. In August, I wrote a script which would alter our database tables to MyISAM one table at a time. Upon launching this script, it took 181 hours (7 days, 13 hours) to process. This process was done with the forums online as 7 days offline was unacceptable. Good thing was, not a single person noticed that we were doing this as it caused 0 downtime.

The outcome of this was noticed immediately. Our database server went from needing at least 16GB RAM to needing only a dismal 4.50GB. Furthermore, database restarts went from taking 45 minutes, or longer, to only seconds as MySQL no longer needed to read through a InnoDB data file that was hundreds of gigabytes in size.

Ultimately, this change was absolutely necessary. Had we not made the change, then today we would need at least 128GB of RAM on our MySQL servers to stay online. Come a few months from now, and we’d be up to 192GB which is currently the maximum limit than any Intel server can address (AMD is limited to 128GB).

Single Database for All Forums

Originally, a single database to house all forums was viewed as a decent choice as there weren’t many forums, there weren’t many tables, and this appeared to be the most simple system to work with. At the time, this was true. But with growth comes new needs, and things change.

As of July 21, there were just over 44,000 forums being hosted by FreeForums.org, and they were all hosted within a single database. This introduces a handful of problems. First, MySQL had to be able to address and stat every single file within the database (over 5 million files) every time it started up. The problem there was that the Linux kernel can only handle so many files in a single directory before it runs into problems, and we were well past that point. There were no inodes left on the hard drive.

The solution to this was implemented while migrating forums from the old database server over to the new database server. Store 2000 forums in a single database, then make a new one and start over. This fixed many of our issues including dumping MySQL data, kernel issues, and helped reduce start time as well as many other issues.

The Future

These changes were the first of many that were needed and still are needed to bring FreeForums.org into a new day; a day free of problems.

If you have any questions, please feel free to leave them in a comment. I’d be glad to answer them.

In the next blog entry, I’ll detail what we have done since July 21 through today.

Hello all!

As I have been saying in the support forums over the last couple weeks, I will be posting blog entries outlining FreeForums.org and its infrastructure of the past, present, and future. But before I can do this, I should really introduce myself to all of you.

I am known as Cero, and I was brought onto the FreeForums.org tech team back in late July as a consultant. I am a programmer and systems administrator with several years of experience with PHP, Perl, shell scripting, as well as XHTML, CSS, and various other programming and markup languages. I currently operate over 35 Linux servers (Debian, Redhat Enterprise Linux, and Ubuntu) as well as a small number of Windows Server 2003 servers.

I have been brought onto the FreeForums.org team in the capacity of a chief technical officer. In that, I now lead the development and changes to the infrastructure (which includes all those pesky upgrades) as well as provide guidance and direction for the programming team.

Lately, many people have been complaining about the upgrades and asking us to stop for awhile. Sadly, that is impossible as it would leave us offline, permanently. So in the coming blog posts, I will be outlining exactly what you want to know and everything there is to know about the infrastructure that FreeForums.org uses to provide you service.

Anyway, if you have any questions you would like to see answered in blog posts, please reply to this and let me know!