As a forum admin, nothing is more frustrating than having your forum spammed to bits. Whether it’s our zero tolerance policy against spammers, proprietary spam prevention systems, or our actively monitored abuse email account, we strive to delete spam before it gets on your plate. We have a goal of removing all automated spam from our network, and have developed proprietary systems that let you enjoy a less spammy diet.

Our systems are so effective that it’s not uncommon for a forum to move off freeforums.org and get bombed with spam the minute they leave. Little is known about our anti-spam systems, and while the technical side is a trade secret (we don’t want the spammers to read this) just know that we are constantly maintaining and updating it to block the most recent spam tactics.

Spam is something that happens and no real way around it. What does matter is how your host handles it.

As our customers know, July & August were huge months for us development wise. In just two months, we made more improvements and installed more features than most hosts install in a years time. Our constant upgrades are a big reason why we’re so far ahead of our competition. Just two months ago we promised five huge improvements were coming this summer. These weren’t small mods that would take a day to install, this were core changes. I’m happy to say that we’re ahead of schedule and already completed four, plus many other exciting features. Here’s an inside look at the coming upgrades:

Free Upgrades — Everyone wants a domain name, and a premium forum, but not everyone is able to pay for it. Coming this month we will allow admins and their users to fill out offers by our sponsors and receive free upgrades.

Forum Directory – This will be a great way to not only find other interesting forums, but also to get users for your forum. Within a few weeks we plan to release a directory with some of the forums freeforums.org hosts. While this directory is limited to really active forums, and premium forums, anyone can purchase premium and get a listing.

Core Network Upgrades – Coming around the 23rd of this month we will be upgrading all three web servers to the latest, more powerful hardware. We will also start upgrading every mysql database server. The result will be even faster page loading and more reliability.

Backup Optimizing – No one would disagree that backups are important. At FreeForums.org, we have one of the most advanced backup systems available. Next week will be removing deadweight files and forums to help optimize the process. This, coupled with the network upgrades mentioned above will highly reduce the slowness some forums see early in the more during the backups.

Knowledge Base Refresh – Our knowledge base is one of the premier help resources for phpBB forums. It’s used by thousands of forum admins per day (a lot use other hosts) and contains a lot of valuable information on starting, managing, and improving your forum. We plan to do a huge refresh this month adding even more content.

As you can see the FreeForums.org team is dedicated towards keeping our service #1, and have quite a bit of work ahead.

If you’ve ever posted a question on our support forum, you’re likely to get helped by one of our volunteer support staff. These users have volunteered their time to help others with questions and problems regarding our service. We only hire the best, most knowledgeable people to ensure our support is top notch. Although we get 10-20 applications a month from those wanting to join our team,  little is known about what we look for when hiring. Below are a few specifics we want in support staff:

• Activity
  It goes without saying that we only want the most active staff. Staff should enjoy helping users and be around to post 10+ times a day.
  
• Knowledge/Quality
  It’s important that staff are well informed and experienced with all of the freeforums.org features and services. This should be reflected in their quality support posts. We’re not looking for people to be correct 100% of the time (we’re all human) but being able to give accurate answers most of the time is needed.
• Politeness
  On the support forum we deal with a lot of users, most are having some kind of problem. When was the last time you called up your Internet provider and told them how much you enjoy their service? Exactly…probably never. The same is true on the support forum. Even though we’re a 100% free service, and we pay a lot out-of-pocket to provide free forums, users expect the best. Being able to work with these users without blowing up is key
• Dedication
  Being loyal and dedicated to FreeForums.org and our community is huge. We need staff who will stick with us through anything, the good or the bad. Who will always put the community first, and remember that the whole purpose is to have fun!
  

Overall the support team is a great way to improve yourself as a person while providing a service to others. As a huge provider of forums we’re great on resumes, and for building your professional skills. If you’re interested in applying please read our application topic!

We get a lot of questions about how we deal with inactive forums. In the past it’s been our approach to keep all forums created until they are removed by the forum owner. While this certainly makes us look good, it can create problems as we continue to grow. Not only do inactive forums take up valuable disk space, but they also extend the time it takes to make backups or install upgrades.

In an effort to improve performance for active forums, as of last month we started running inactivity audits. If your forum goes 60 days without a post, we will email you letting you know your forum is about to expire. You will have 7 days to make a post on your forum or it will be permanently removed from our system.

If you get this email and want to save your forum, simply make a post anywhere on your forum. Having said that, if you fail to do so and it’s removed after 7 days, we are unable to recover it.

Our active forums will be very pleased with this change as it will improve the level of speed and reliability you have come to love and expect.

When it comes to development, we have always released features on a “when ready” basis. This often left a limited time for testing and users unsure when the next upgrades would take place. Effective today, we will be switching to a two week development cycle.

What does this mean for you? You can now expect new upgrades to take place around the 1st and 16th of the month. We’ve included a diagram below to explain how this works behind the scenes:

We hope this set schedule allows more transparency on our development process, and gives you more insight into when features and mods will be released. It will also allow us to test features more before they are released on our production server.

Forum hosting today is a very competitive market and hosts must fight to survive. One of the advantages we have is our collection of modifications or mods for short. Mods alter your forum in someway, whether it be a small quick reply or large chatroom. When FreeForums.org launched in February 2007 we changed the game forever with our constant upgrade policy. Today, I’m going to outline some of the work that goes into bringing a mod off the shelf and onto your forum.

We are always looking to provide our customers all the mods they need to run a successful forum. Everytime we evaluate a mod we run it through a few questions. “Is this what our users want, will it fit with our mission of what a forum should do?” Once we decide to install a mod it goes over to our technical team where we discuss security and scalability issues. Because of our proprietary systems and the need for mods that meet our high standards, we almost always end up coding our own version of the requested mod.

After research and development we move on to testing. We never rush anything into production and want to be sure that it really does preform the way we intended. This step also includes making sure every mod works on all 350+ templates between v2 and v3. With mods being installed on over 300,000 different forums there is no room for error.

In 2009 with will be highly focused on mods for our v3 system so please let us know what mods to add!

With the continued work on our phpBB3 system taking up the majority of our development time some users have emailed us asking if we even care of receiving feedback.

Before we get started I’d like to make it clear that we not only value user feedback, we thrive on it. Our great success today is in part due to the innovative ideas that come through our users. Having said that, if you’re a user on the support forum the feedback you see is in some ways deceiving.

We have over 10 Million active users on our network of forums, compared to just 8,000 on the support forum that posted in our feedback board. That means 0.001% of our total users are actually on the support forum offering feedback, a very small number indeed. To put that in perspective even more that’s like speaking to 100,000 users and hearing feedback from 1 person. It also so happens that the most active forums and users aren’t the ones leaving feedback on the support forum. It tends to be the forums just starting up rather than the already established ones.

So how do we gauge feedback? The best way we can: Data. We collect huge amounts of data that we use to determine which features people actually use and like. We use this data not only to improve on existing features, but also new ones we’re looking to offer.

Some of our most successful features (ie. Community Chest and Welcome box) have taken a hit on the support forum by a few lone users, although we were able to see by the data that they were highly welcomed by the masses.

In the end, next time you see 10 users asking for a feature or complaining about something on the support forum don’t be so quick to assume all users hold those views. We will watch closely and if the voices are in tune with the masses, we will take action. What we won’t do is let a group of vocal users make the decisions felt by millions.

Since July 21, focus shifted at FreeForums in the way we thought and went about seeing our future. We wanted to make sure that what happened that night could never happen again. At the same time, we were seeing problems brewing in the air.

Database Clusters
Due to the rate at which FreeForums was growing and the limitations of MySQL replication, we saw ourselves running into a very severe road block. To maintain our MySQL servers, at the time we had one master MySQL server and several slave servers. All writes would go to the MySQL master which would then be synchronized to the MySQL slaves which all reads went to.

The problem we saw was that MySQL replication, on the slaves, was only capable of using a single CPU core to process updates coming from the master. To handle the operation of replication, the master server needed use of all its cores, yet each slave could only use one. Eventually we would hit a wall at 100% usage of the core and thus be unable to process any more updates. The result from this: slaves would fall out of sync.

Our solution to this was to break FreeForums up into multiple independent MySQL clusters. This would distribute writes over serveral master servers and allow us to add even more slave servers. But writing a system for this would not be easy and would take some extensive testing and development.

All the while we were developing this new system, we began hitting the writing bottleneck on our slaves. Throughout December, we were having slaves fall out of sync whenever there was high traffic hitting our web servers; however, thanks to the holidays, we were given two weeks of lower traffic which gave us more time to test and develop the system. Come January, however, that went away and for the entire first week and a half, slaves were out of sync every day from 11AM thru 7PM.

Given the obviousness that we were out of time, we halted testing of the second cluster system and launched it. The tests had been going well, and we wanted to continue testing a few more things to make certain that the system would work as expected, but we were out of time. On the morning of January 15, while watching Cluster-0 slaves fall out of sync at 6AM, I decided that we had to launch the system immediately. At that time, we picked 39 databases accounting for 20% of our daily traffic and sent them to the new cluster.

Low and behold, the Cluster0 slaves synced back up, and since that date we haven’t had a single slave fall out of sync. At this time, we have finished testing and development of the clustering system and will be using it from now on. The addition of clustering was a huge step forward for us as it will guarantee the speed and stability of FreeForums in the coming years.

Web Servers
The second problem we were already facing was that we needed a more stable environment to serve the websites that we were hosting. We had multiple web servers, but the way in which we were load balancing them was by the use of round robin. Round robin is a method where in your DNS records, used by the Internet to look up where your website is located, you list multiple IP addresses. When you type in your web address, your computer randomly selects from that pool of IP addresses and that is the server you land on. Problem here is that there is no load distribution and there is no failover protection.

Those two issues would spell chaos as FreeForums grew. Even though we had three web servers, one of them was receiving 50% of all traffic with the other two receiving 25% each. Next, what would happen if, say, the web server receiving 50% of traffic would go down? The answer is simple: 50% of viewers wouldn’t be able to access the forums they were trying to view.

The solution to this was simple, in theory; however, it would take time to test and develop. Consulting with SoftLayer network engineers, we were directed to look at the Zeus ZXTM Load Balancer. Zeus ZXTM is a software suit that distributes requests for various TCP and UDP services based on specifications that you set for it.

What Zeus ZXTM would give us is a level of stability and redundancy on our web servers that has never been seen before my FreeForums.org and its users. By sending all requests to the Zeus ZXTM load balancing server, we are able to evenly distribute all requests over all web servers as well as redirect requests when a web server goes offline. This will make it impossible for a failed web server to ever be noticed by our viewers again.

As of January 26, all viewers are accessing FreeForums.org by way of the Zeus ZXTM load balancer, and we couldn’t be happier.

The Future
FreeForums.org is well on its way to making all points of service HA (high availability) by making every single service redundant with backup servers, but we still have a ways to go before we achieve this.

First, the global MySQL servers used to house the data shared between all clusters, while having a slave keeping mirrored copy of the data, is still setup in a Master+Slave method. Should the master go down, a human will need to convert the slave to a master server. The solution to this is to utilize MySQL Cluster, which allows for all servers in the mix to be masters. Once this is done, if the primary server ever goes down the others can take over immediately without anyone needing to intervene.

Next, the file storage server is currently standalone. There is no failover, there is no mirror. Just backups being made every fifteen minutes. But, aha!, yet another easy fix. We will be looking into using the Lustre File System as a means to create several file servers which act as one unified server. Should a server ever fail, go offline, or do anything that makes it unavailable, all requests to that server will be directed to the others. No one would ever know the difference.

Lustre is based off Sun Microsystem’s ZFS, a very powerful and robust filesystem. Lustre is currently in use by several of the world’s largest supercomputers, hosting hundreds of millions of files totaling petabytes in total space. So we’re confident that it can handle our humble hundreds of thousands of files.

Many more changes are coming in the future, and things are sure to be very exciting as we move into the future.

Lately we’ve been getting a lot of questions about the next big thing from FreeForums.org. Although we haven’t been adding much to our constant upgrade topic, we have been working very hard on what we like to call “operating features.”

Operating features are features that indirectly improve your forum. They are necessary to keep the FreeForums.org Efficiency and are mostly on the back end. This include things like tweaking our proprietary spam prevention systems, streamlining the support forum, and optimizing the current programs.

We do have some exciting new features in the works so keep checking back.

FreeForums.org is known for the low overhead costs associated with being efficient. Efficient severs, efficient code, efficient work schedules, all the way down to efficient use of resolving topics. We’re always looking for ways to shave seconds off the time it takes to complete a task. Which is why we recently added the “Self Resolve” button to all support posts. Now, after fixing the problem or getting your answer, you can mark your own topic resolved.

We found that over 40% of topics are resolved by the time staff get the chance to resolve it themselves. Taking the 100+ tickets a day we see, that’s 40 times a day our staff don’t have to enter a thread, read through the problem, see it’s been resolved, and click the resolve button. Estimating 20 seconds per thread, this saves us 14 minutes a day, 1.5 hours a week, and 6 hours a month!

Help us help you, resolve your own topic!