NFS buffer issue? ZFS issue? nginx?

R

ripken204

[H]ard|Gawd
Joined
Aug 24, 2007
Messages
1,427
I have two AIO ZFS builds, very similar hardware.
ESXi 5.0, OI as the SAN
Ubuntu 10.04 as a web server which mounts the array from the SAN via NFS
I am using this command in my fstab:
192.168.0.252:/storage/www /www nfs rsize=8192,wsize=8192,timeo=14,intr
Click to expand...

The issue I've been having with both of these servers is delay from when I press save in my text editor until when I see the changes on my website. I have nginx + php-fpm running, the sites are mostly PHP, using CodeIgniter. The server are meant for local dev.

At first I thought the issue was nginx or php-fpm but I can't figure it out.
To describe my issue, if I save my code, go to my browser and press F5, sometimes the new code renders immediately, sometimes I have to wait anywhere from a few seconds to thirty seconds. This is incredibly annoying, especially when I assume that I am running code which I hadn't planned on running.

I don't have any sort of cache enabled, this is a very clean LEMP install on a very clean OS install. The stranger thing is that this occurs on two different servers which are in two different locations. The servers are only being used by one or two people at a time, very little load.

Could NFS be caching or buffering the files?
Any ideas on what could be causing this issue?
I suppose my next step is to try mounting it as a smb share...
 
here is what you must do.

create a local copy of the site on the ubuntu machine so it is local to that filesystem. make a change. see if the webserver updates the change. basically take the nfs out of the picture. make sure it's the exact same setup, codeignitor and all.

if the change is not immediate, your php setup or nginx setup is caching something. if the change is always immediate, your nfs is caching something.
 
Is the client using nfs sync mode? You can test this by setting sync=disabled on the pool and re-test.
 
Ubuntu is a VM which is stored on the ZFS partition, so ZFS could be the issue too. But you're right, that would eliminate nginx.
I set sync=always, I will do some testing over the weekend and see if that fixes it.

It's not a browser issue, I have ctrl+f5'ed in Chrome,FF,IE on different computers on different networks and different users.

I just added sync to my fstab on the client side. Between setting that and ZFS sync=always, it will hopefully work... Assuming ZFS or NFS is the problem.
 
Last edited:
well that hasn't fixed the problem.
next step is to host the files on the ubuntu vm, which is still on the ZFS array but at least there won't be NFS involved.
 
Confused why you wouldn't check the state of the file from Ubuntu command line...thus eliminate BOTH potential web server and browser as issues.

My best guess is that you are not time-syncing your all your systems (including the one where browser is). If browser is caching and then issuing if-modified-since: **a datetime** I'm thinking your cache not get refreshed if there are time sync issues between the involved hosts (this is just a guess as I haven't ever tested this).
 
I might test via the terminal.
I mounted with CIFS and it seems to be working fine right now.

What exactly do you mean by time sync? the actual system time?
Ubuntu was correct, OI was a few minutes slow
I just ran ntpdate ntp.ubuntu.com on both servers and now they both have the same time.
 
Granted I'm not using your setup or OS, but with the NFS share that I'm using, I have it set to ASYNC, and honestly, transfers go much faster since it only needs to recognize the files in place, rather than the exact amount of data being confirmed constantly, thus bogging down a connection.

This may not be your problem, but it could be a start.
Did you test it with sync disabled?
 
I haven't tried that because the description in napp-it makes it sounds like I wouldn't want to use it

Synchronous requests are disabled. File system transactions
only commit to stable storage on the next DMU transaction group
commit which can be many seconds. This option gives the
highest performance, with no risk of corrupting the pool.
However, it is very dangerous as ZFS is ignoring the synchronous
transaction demands of applications such as databases or NFS.
Setting sync=disabled on the currently active root or /var
file system may result in out-of-spec behavior or application data
loss and increased vulnerability to replay attacks.
Administrators should only use this when these risks are understood.
Click to expand...
 
It depends on what you do and your setup. If you have an all-in-one, I think sync mode makes no sense. If it is not an all in one, it depends on what kind of writing you are doing. As it says, you need to assess the risk. Keep in mind that most filesystems do all kinds of async writes from the applications, so I'm not convinced this is not overly alarming.
 
the zfs folder in this case is purely for web development, no heavy reads/writes.

I haven't had an issue since syncing up the time, hopefully it stays that way.
 
I moved the www folder to the ubuntu (webserver) VM and the speed is good...
It has to be an issue with connecting to OpenIndiana's via NFS, still not sure what the problem is though.
I have also tried the async parameter for mounting nfs in fstab, still slow

When I mount with CIFS it seems to work fine, but file permissions are screwy. I mount the drive as root user.
For example, when I try to git commit on a directory, it throws a permisson denied error on .git/index.lock
 
Last edited:
ripken204 said:
I moved the www folder to the ubuntu (webserver) VM and the speed is good...
It has to be an issue with connecting to OpenIndiana's via NFS, still not sure what the problem is though.
I have also tried the async parameter for mounting nfs in fstab, still slow

When I mount with CIFS it seems to work fine, but file permissions are screwy. I mount the drive as root user.
For example, when I try to git commit on a directory, it throws a permisson denied error on .git/index.lock
Click to expand...

Solaris CIFS = Microsoft compatible = ACL only
set ACL of your shared folder like everyone@ or connected user to at least modify-set with inheritance=on
and it should work as desired

NFS V.3 on the other hand is Unix permission only based on hosts not on users.
If you compare ZFS vs Ubuntu/ext3, be aware that ZFS is slower but with better data security on single disks but usually scales better
with amount of vdevs/ raid-sets. It is a technology developed for best security and performance with a lot of disks
 
Last edited:
how is this done through napp-it? if possible at all?

on the ZFS Folder tab under FOLDER-ACL (O,G,E) I have o=full, g=full, e=full
under SMB-SHARE-all I have full_set
under PERM I have 777+
RDONLY = off

I can mount the share via CIFS for my user and root user, but neither my user nor root has permission to write.
I even tried changing my local mount point to /home/myuser/www and I still don't have permission to write

where is the option for inheritance?

to sum up what I want to do:
I have a /storage/www ZFS Folder on OI
In Ubuntu (my webserver I have /www which mounts /storage/www, I am trying to use CIFS right now, NFS was giving me problems.
I have multiple users on the machine, each with their own accounts, all of them need full access to /www
 
Last edited:
ripken204 said:
how is this done through napp-it? if possible at all?

on the ZFS Folder tab under FOLDER-ACL (O,G,E) I have o=full, g=full, e=full
under SMB-SHARE-all I have full_set
under PERM I have 777+
RDONLY = off

I can mount the share via CIFS for my user and root user, but neither my user nor root has permission to write.
I even tried changing my local mount point to /home/myuser/www and I still don't have permission to write

where is the option for inheritance?

to sum up what I want to do:
I have a /storage/www ZFS Folder on OI
In Ubuntu (my webserver I have /www which mounts /storage/www, I am trying to use CIFS right now, NFS was giving me problems.
I have multiple users on the machine, each with their own accounts, all of them need full access to /www
Click to expand...


- the + in 777+ indicates, that there are ACL's
- You can check or set ACL via Windows (connect as root) or nappit menu extension-acl
- ACL settings of the folder does not affect already created files
ex reset ACL from Windows with inherit settings to file/folders
 
my issue seems to be with mounting it
I think I am close, I needed to add file_mode and dir_mode
I can edit files from multiple accounts now

\\192.168.0.100\www /www cifs username=root,password=mypass,dir_mode=0777,file_mode=0777,_netdev 0 0ir_mode=0777,file_mode=0777 0 0
Click to expand...

I am still having issues when I commit a git repo
error: unable to create temporary sha1 filename .git/objects/0f: Operation not permitted
Click to expand...
 
ripken204 said:
my issue seems to be with mounting it
I think I am close, I needed to add file_mode and dir_mode
I can edit files from multiple accounts now

I am still having issues when I commit a git repo
Click to expand...

my opinion

do not think Linux
Sun did its best to make CIFS working like Windows
You must use Solaris CIFS like you would use a Windows server: ACL only

set everyone@=modify with inheritance=on to the shared folder for newly created files
or reset this permission (ex from Windows) to such a permission for already created files (ex from Windows)

And yes: I know, there are problems to emulate Windows behaviour in Unix - the reason for these problems
 
Last edited:
damn windows...

for my git issue, I just deleted my .git folder and then git init again, all is well now
permissions seem to work as I stated in my last post
hopefully I have no issue with speed with CIFS, it seemed to work fine in my brief testing of it

thanks Gea, you are a real help to all of us
 
ripken204 said:
Ubuntu was correct, OI was a few minutes slow
I just ran ntpdate ntp.ubuntu.com on both servers and now they both have the same time.
Click to expand...
Having correct system time on networked computers is one of the most basic admin stuff, especially when having shared filesystems. And ntpd > periodic ntpdate.

DNS comes next IMHO.

Always check those before doing anything else.
 
i am working off of ip address so DNS wont be an issue. I will set up automatic ntpd.
I am a wannabe sys admin so i am still learning, thanks.
 
just an update: CIFS is working perfectly. NFS was the issue, I am still not sure why.
 
You must log in or register to reply here.
Back
Top