Modify

Ticket #986 (new task)

Opened 3 years ago

Last modified 3 years ago

Remove Nginx?

Reported by: anna Owned by:
Priority: major Milestone: 4.6
Component: Architecture Severity: New Ticket
Keywords: Cc:
Who will test this: And

Description

Sam has suggested some ways to think of removing Nginx from the Plumi stack, and rely upon Apache alone.

I've added all his notes here.

In general EM might want to consider simplifying their server architecture. Plone with just Apache server in front of it, and a disk cache configured in Apache, can serve busy sites very nicely ... this one for example that we manage:  http://www.ethicaljobs.com.au/ (see somewhat dated visitor stats here:  http://www.ethicaljobs.com.au/news/ethicaljobs.com.au-celebrates-115-000-visitors). An additional advantage is that the buildout is simplified quite a bit too - no nginx, varnish, ...

On 11/12/14 11:58 PM, Markos Gogoulos wrote:

We had a really bad time with timeouts so after research we ended up in nginx + nginx upload module (before that we were on Apache). Even if 2-3G file uploads are supported by apache/nginx (which I doubt), the combination with uwsgi makes things much complex.

In fact the nginx upload module uploads files in a /tmp dir, and when the whole file is uploaded it handles it to zope, that moves it in blobstorage etc.

Nginx is used so that large files can be uploaded (we have tested with up to 3G files), we are using an nginx's plugin for this. Without this plugin, apache/nginx timeouts after some minutes of video uploading.

On Thu, Dec 11, 2014 at 1:57 PM, Sam Stainsby <sam@…> wrote:

On 11/12/14 19:41, Markos Gogoulos wrote:

Nginx is used so that large files can be uploaded (we have tested with up to 3G files), we are using an nginx's plugin for this. Without this plugin, apache/nginx timeouts after some minutes of video uploading.

Any recent mainstream web server (Apache, Nginx, ..) should handle large (or long) file uploads just fine. Are you sure it's not a proxy timeout you're seeing?

Cheers, Sam.

I can think of at least three multimedia projects I've build that can do this with plain Apache, so I suspect your timeout problems lies with what runs behind your web server rather than the web server itself ... or with bad proxy settings ... or just a bad network, in which case a chunked, resumeable uploader Javascript library could be the solution.

In fact, the last system I built with file uploads wasn't designed with large files in mind, but out of curiosity I'm testing it right now with a 2.3GB upload. It's been running fine for 1.5 hours so far. Unfortunately, at my ADSL1 network speed (current upload progress is at 7%!), it will probably take all day ... possibly more. As it's taking up too much upstream bandwidth, I may have to stop it soon.

If interested, below are some technical details ...

The Apache virtual host is very plain and boring:

<VirtualHost? 106.187.99.62:80>

ServerName? admin.realbreakthroughs.net CustomLog? ${APACHE_LOG_DIR}/admin.realbreakthroughs.log combined ErrorLog? ${APACHE_LOG_DIR}/admin.realbreakthroughs.error.log DocumentRoot? /var/www/realbreakthroughs/admin_ui <Directory />

Options FollowSymLinks? AllowOverride? All

</Directory> <Location />

AuthType? Basic AuthName? "RB restricted area" AuthUserFile? /etc/apache2/realbreakthroughs_password Require user rbadmin

</Location> RewriteEngine? On RewriteRule? /api/(.*)  http://localhost:3122/$1 [P,L,E=proxy-timeout:200s]

</VirtualHost>

  • the front end uses angular-file-upload.js and the back end behind

Apache is a RESTful node.js API server. It streams uploaded files directly into MongoDB GridFS using about 80 lines of code. Seriously, this is the upload handing code in the node.js part:

exports.insert = function(db) {

var gridfs = GridFS(db, mongodb); return function(req, res) {

var form = new multiparty.Form();

form.parse(req, function(err, fields, files) {

if (err) {

res.status(500); return res.send(err);

} try {

var metadata = JSON.parse(fields.metadata); var fileKeys = Object.keys(files); var nKeys = fileKeys.length;

if (nKeys < 1
nKeys > 1) {

return res.send(422, 'wrong file count: ' + nKeys);

} var file = files[fileKeys[0]][0]; var filename = file.originalFilename;

var ctype = file.headers && file.headerscontent-type?

'application/octet-stream';

var writeOption = {

filename: filename, mode: 'w', content_type: ctype, metadata: metadata

}; var tmpPath = file.path; var outstream = gridfs.createWriteStream(writeOption); outstream.on('error', function(err) {

console.log('ERROR: FAILED storing file'); return res.send(500, err);

}); outstream.on('close', function(data) {

return res.send(201, data);

});

var instream = fs.createReadStream(tmpPath); instream.pipe(outstream);

} catch(err) {

console.log('UPLOAD ERROR: ' + err); res.status(500); return res.send(err);

}

});

};

};

Cheers, Sam.

Attachments

Change History

comment:1 Changed 3 years ago by sam

Moreover, consider removing nginx, varnish and uwsgi and just having Apache (which can do caching and load balancing quite nicely these days) proxying clustered Plone.

View

Add a comment

Modify Ticket

Action
as new
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.