Today we released a new feature that allows board owners to enable In-Text ads on their forum and receive service credits for doing so. We understand that while enabling ads is a big plus for you, some users may think otherwise. Because of this, we’ve prepared some points you can relay to your members:

• These ads will benefit them by allowing you (the admin) to purchase additional features to improve the forum.
• They’re not intrusive like popup ads or flashing banner ads.
• They can disable them if they want by clicking the question mark in the ad box. However, doing so will hurt your forum so this isn’t recommended.
• They’re relevant to the information on the post and can actually be informative.
  

Last week we started asking users to submit their forum story and we would like to thank everyone for their submission. I’m pleased to announce the first featured story:

My Freeforums site started out as a forum for hacking numerous online video games like Socom. That only lasted the first month or so though. I decided we should cover gaming news in general so over the next few weeks we added console sections for new, reviews and tutorials. Now we are a rapidly growing gaming, emulation and hacking forum (with nothing illegal of course) with over 1100 members!

People use the site for news and guides on how to do anything from playing Mario on their psp to how to speed up their computer.-Shane Craig
www.Codemayhem.net

Codemayhem.net will receive a refund on their already purchased domain or another domain if they choose. This is an ongoing thing so it’s not to late to submit your story and get entered to win a free domain!

To submit your story go to http://www.freeforums.org/story.php.

During the brief outages we have from time-to-time, we are constantly reminded how important our users’ forums are to their everyday lives. We love to hear feedback on our services but above all, we love to hear how our users are using their forum.

This is the reason behind our new FreeForums.org Storyboard. Please submit your story, including how you use your forum and how it benefits your members. We will print off each story and tape it next to the giant whiteboard we use to map out new ideas or plans. Every month for the next 3 months we will pick one off the whiteboard, post their story (free advertising for your forum!), and give them a free domain.

To submit your story please go to the FreeForums.org Storyboard page. Remember, this isn’t a testimonial on FreeForums.org service but rather a story on your forum and how you use it. Feel free to get creative and make it as long as you want. If you would like to upload a video to youtube, there will be extra prizes

Before I begin, I should note that this is a very technical and in-depth post. Good luck.

FreeForums.org launched in February of 2007, and has grown at a rapid pace ever since.  With that comes growing pains.

For the first many months, a very simple set was required to keep the forums moving along. Within a few months, the need for more power came, but still nothing a single Intel Xeon couldn’t handle. But that was bound to change. At the time that I came to FreeForums.org, the setup was as follows:

Web Server
Single Intel Xeon L5310 (quad-core, 1.60Ghz, 8MB L2 cache)
2GB DDR2-667 FBDIMM RAM
250GB SATA2 Hard Drive
Overall CPU Usage: 30%

Database Server
Quad AMD Opteron 8212HE (dual-core, 2.00Ghz, 1MB L2 cache)
16GB DDR2-667 FBDIMM RAM
250GB SATA2 Hard Drive
Overall CPU Usage: 20%

At the time there were several major flaws in design that originally did not pose a problem.

Non-RAID Data Storage
In the world of servers and corporations, data redundancy and protection is an absolute must. What RAID does is it takes multiple hard drives and configures them in such a way that the loss of a numbre of hard drives (due to failure, for example) will not result in data loss (this is true for all levels of RAID except for RAID0).

The importance of RAID became very obvious the night I began consulting for FreeForums.org. That night, I was called in because all attempts to syncronzie the new MySQL server with the existing server’s data failed. The InnoDB database (see below) was unable to be read by the new server no matter what the staff tried to do.

My solution was that if we couldn’t move the data file, then let’s export the data ourselves and send it over row-by-row to the new server. But there laid yet another problem: All of the tables were in a single database. MySQL’s SQL export utility “mysqldump” was unable to dump SQL data as fast as we needed it to (45 minutes per forum at 44,000 forums or so, total time: 3.77 years). So my solution was to write a script that would read the data from MySQL, create the proper schema for it, and send it to the new server.

On the night of July 21, while the script was running and moving forums (it had moved roughly 4000 forums), the old MySQL server crashed and would never come back up. The single hard drive housing the InnoDB data file (containing all the forum data) corrupted and failed. After trying for 14 hours to recover the data, it became obvious that the data was lost.

The solution to prevent this in the future was simple: utilize RAID. The first hardware change I made with FreeForums.org was to institute the usage of RAID10 on all MySQL servers. RAID10 is a level of RAID which utilizes the speed benefits offered by RAID0 but provides data protection at the same time. In RAID0, each hard drive is setup in a linear segment in which data is sent in blocks across all the hard drives. It allows you a very fast virtual hard drive, but it offers no data protection. Four 80GB hard drives setup in RAID0 will offer you 320GB of storage space, but if even one hard drive fails you will lose all your data. RAID10 allows you to gain the benefits of RAID0, but it gives you data protection by pairing up each segment with a RAID1 mirror. In this case, four 80GB hard drives will offer you 160GB of storage space, but if you lose a drive in any segment, you’re safe. You can lose one drive in every segment without data loss, but should two drives in the same segment fail, then you lose all of your data. But the chances of that happening are slim to none without you having enough time to fix it.

Another benefit of RAID is that it gets you added disk bandwidth. The more drives you have, the more bandwidth is available to you. For us, that is one definite plus.

Thanks to RAID, we will never have to worry about what happened on July 21 from ever happening again.

InnoDB Storage Engine in MySQL
InnoDB is a great storage engine when you have very large and active tables, but when you have an insane amount of tiny tables, InnoDB becomes a huge problem. Mainly, InnoDB uses a lot of system resources to read data from it’s main file (ibdata1). Back in July, we needed, at an absolute minimum, 16GB of RAM on the MySQL server to stay online for even 8 hours. But the truth of the matter was, we needed 32GB to remain stable, and that number would only grow as FreeForums.org grew.

The solution to this was a simple yet time-consumng task. Convert from the InnoDB storage engine and to MyISAM, which is more suited to our needs. In August, I wrote a script which would alter our database tables to MyISAM one table at a time. Upon launching this script, it took 181 hours (7 days, 13 hours) to process. This process was done with the forums online as 7 days offline was unacceptable. Good thing was, not a single person noticed that we were doing this as it caused 0 downtime.

The outcome of this was noticed immediately. Our database server went from needing at least 16GB RAM to needing only a dismal 4.50GB. Furthermore, database restarts went from taking 45 minutes, or longer, to only seconds as MySQL no longer needed to read through a InnoDB data file that was hundreds of gigabytes in size.

Ultimately, this change was absolutely necessary. Had we not made the change, then today we would need at least 128GB of RAM on our MySQL servers to stay online. Come a few months from now, and we’d be up to 192GB which is currently the maximum limit than any Intel server can address (AMD is limited to 128GB).

Single Database for All Forums

Originally, a single database to house all forums was viewed as a decent choice as there weren’t many forums, there weren’t many tables, and this appeared to be the most simple system to work with. At the time, this was true. But with growth comes new needs, and things change.

As of July 21, there were just over 44,000 forums being hosted by FreeForums.org, and they were all hosted within a single database. This introduces a handful of problems. First, MySQL had to be able to address and stat every single file within the database (over 5 million files) every time it started up. The problem there was that the Linux kernel can only handle so many files in a single directory before it runs into problems, and we were well past that point. There were no inodes left on the hard drive.

The solution to this was implemented while migrating forums from the old database server over to the new database server. Store 2000 forums in a single database, then make a new one and start over. This fixed many of our issues including dumping MySQL data, kernel issues, and helped reduce start time as well as many other issues.

The Future

These changes were the first of many that were needed and still are needed to bring FreeForums.org into a new day; a day free of problems.

If you have any questions, please feel free to leave them in a comment. I’d be glad to answer them.

In the next blog entry, I’ll detail what we have done since July 21 through today.

Hello all!

As I have been saying in the support forums over the last couple weeks, I will be posting blog entries outlining FreeForums.org and its infrastructure of the past, present, and future. But before I can do this, I should really introduce myself to all of you.

I am known as Cero, and I was brought onto the FreeForums.org tech team back in late July as a consultant. I am a programmer and systems administrator with several years of experience with PHP, Perl, shell scripting, as well as XHTML, CSS, and various other programming and markup languages. I currently operate over 35 Linux servers (Debian, Redhat Enterprise Linux, and Ubuntu) as well as a small number of Windows Server 2003 servers.

I have been brought onto the FreeForums.org team in the capacity of a chief technical officer. In that, I now lead the development and changes to the infrastructure (which includes all those pesky upgrades) as well as provide guidance and direction for the programming team.

Lately, many people have been complaining about the upgrades and asking us to stop for awhile. Sadly, that is impossible as it would leave us offline, permanently. So in the coming blog posts, I will be outlining exactly what you want to know and everything there is to know about the infrastructure that FreeForums.org uses to provide you service.

Anyway, if you have any questions you would like to see answered in blog posts, please reply to this and let me know!

Lately we’ve been getting a lot of questions about the next big thing from FreeForums.org. Although we haven’t been adding much to our constant upgrade topic, we have been working very hard on what we like to call “operating features.”

Operating features are features that indirectly improve your forum. They are necessary to keep the FreeForums.org Efficiency and are mostly on the back end. This include things like tweaking our proprietary spam prevention systems, streamlining the support forum, and optimizing the current programs.

We do have some exciting new features in the works so keep checking back.

If you’ve been around forums a lot you have probably heard “Don’t bump your topic” over and over. While topic bumping is considered spam and looked down upon, there is a good time and place for it.

As part of our goal to offer you the best possible support, we’re pleased to announce the “Auto Bumping” system on the support forum. Effective immediately, when a new thread goes over 24 hours without a response, it will be automatically bumped to the top and flagged for admin review.

This is just one of the many upcoming improvements to our support system.

Lets face it, companies are always trying to gain new customers but often forget the most important customers are the ones they already have. This post is dedicated to the customers we already have and why you should continue hosting with us.

1. We’re forum people
With almost all our staff having operated a successful forum in the past, we know what it’s like on the client side.

2. Customizable for your needs
We don’t have a ‘one size fits all’ approach to forum hosting. Rather, we tailor solutions to a forum’s needs, providing you with all the features and options you need to run a successful forum.

3. True 24/7 support
A lot of hosts run “24/7 support forums” but depending on what time you post, they may be sleeping. At FreeForums.org, we strategically hire support techs around the globe to ensure we have a full 24/7 coverage.

4. Features and Upgrades
Those who have been with us awhile know that we’re serious when it comes to adding new features. We are constantly reviewing suggestions, doing market research, and thinking up new and innovative features.

5. All-Star lineup
We knew from the start it would take the best staff in order to create the best forum host. Our staff consists of over 30 years hosting experience.

6. Version 3 will be an upgrade, not a downgrade
Like most other hosts, we were tempted to throw together some half written code lacking structure and features in order to be the first out there with phpBB3. Fortunately for our users, we put them ahead of our desire to be the first. When we release version 3 we want it to be an upgrade, not a downgrade.

7. We actually listen
We don’t simply have a feedback & suggestion board for the sake of doing so, we actually read and consider every post made. Customers come up with great ideas, and we respect that.

8. We’re cool
Ever increasing attach storage? Forum toolbars? You got it! At FreeForums.org we put the “cool” back in forum hosting.

9. Free Upgrades for your forum
Other hosts make you pay for every upgrade you want. Depending on the user, this can make it difficult to obtain that upgrade. We offer a new type of system, “Free Upgrades,” which allow you to earn points completing offers. You can then redeem these free points for an upgrade to your forum.

10. We treat every user’s forum as our own
If you’re having a problem with your forum, whether it is technical or not, we take the situation to heart and provide the best possible support that we are able to in order to help our users.

FreeForums.org is known for the low overhead costs associated with being efficient. Efficient severs, efficient code, efficient work schedules, all the way down to efficient use of resolving topics. We’re always looking for ways to shave seconds off the time it takes to complete a task. Which is why we recently added the “Self Resolve” button to all support posts. Now, after fixing the problem or getting your answer, you can mark your own topic resolved.

We found that over 40% of topics are resolved by the time staff get the chance to resolve it themselves. Taking the 100+ tickets a day we see, that’s 40 times a day our staff don’t have to enter a thread, read through the problem, see it’s been resolved, and click the resolve button. Estimating 20 seconds per thread, this saves us 14 minutes a day, 1.5 hours a week, and 6 hours a month!

Help us help you, resolve your own topic!

At FreeForums.org, we’re always moving at the speed of light. Creating this blog was no exception.

I came up with the idea for a blog only four days ago, and your feedback convinced me to set everything in motion. Here we are, four days later and I’m proud to announce the Inside FreeForums.org blog. We understand the importance of connecting to customers in every way possible. On this blog we look forward to posting the thoughts and opinions of FreeForums.org staff. This blog also gives customers a way to provide feedback that we will read.

We have quite a few exciting posts planned for the near future so keep checking back. Or, you could join the 21st century and subscribe to the RSS feed.