OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
I am seeing the same here is my setup:

1x Supermicro SC846E26-R1200B Chassis
1x Supermicro X8DTH-6F Motherboard
1x Intel Xeon E5620 / 2.4 GHz processor
1x Kingston Memory (3x 4GB) 12 GB Total
10x Seagate Constellation 1 TB SATA Hard Drives

Running ESXi 4.1 then have Nexenta running on SSD Storage device. Nexenta has PCI Passthrough to the SAS Backplane and has 10 Segate SAS 1TB Drives installed running 4 mirrors striped to make up my datastore. 2 of the 1tb drives are mirrored and are shared as a separate datastore being shared to ESXi via NFS for my VM's.

I then installed a windows 7 ultimate VM. Added a iSCSI Drive that had been shared using Nexenta vDev from the 4 mirrored 1 TB drive datastore and ran benchmarks against it using several programs. Crystal Mark is consistently around the 50 MB/s. I feel like I should be much higher than this and can find other posts of people stating they are hitting around 200 MB/s.

Is this typical or am I mssing a setting . Any tips or tweeks. This is all contained inside an all in one box so I hoped for much higher speeds.


pure mirrors or multiple added mirrored vdevs (Raid-10 and better)?

and
-all you need to know
http://constantin.glez.de/blog/2010...ove-oracle-solaris-zfs-filesystem-performance
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide
http://hardforum.com/showthread.php?t=1573272

Gea
 

jwinsor566

n00b
Joined
Feb 26, 2011
Messages
13
I have not disabled Sync. I have tried with a SSD ZIL and without and did not see much of a differance good or bad with it. I think it helped with IO's but not transfer. I will try disabling Sync and ZIL and tell you the results.

Gea,

I will review your links. I followed your tutorial for the most part to configure my all-in-one. I have 4 mirrors added to one Pool. So I believe that is Stripped mirrors.

thanks
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
I have not disabled Sync. I have tried with a SSD ZIL and without and did not see much of a differance good or bad with it. I think it helped with IO's but not transfer. I will try disabling Sync and ZIL and tell you the results.

ZFS caching (RAM or SSD) is ignoring large files.
Thats really very good but it will improve I/O only or access to small files or metadata like directory infomation, not sequential speed with large files.

to improve this, you need spindels, Ram, CPU and should avoid dedup, enryption and compression. For virtual machines, you can use VMware VMXNET3 virtual network driver. If you need real power, use SSD datadrives instead of harddisks.

I have 4 mirrors added to one Pool. So I believe that is Stripped mirrors.

correct
 

DJ_Datte

Weaksauce
Joined
Oct 11, 2010
Messages
89
About the answer "you need spindles", in my example, I have 16 spindles, of fairly new 2TB drives. We all know any of those disks would in windows manage ~100mb, and 2 striped together would almost certenly cap 125mb/sec (1Gbit connection) in simple large file reads.

My (our, I guess) question is, why does solaris/zfs based systems have so much trouble doing it with 16 spindles, again, on simple, sequential large-file reads ?

And, why, when non-networked performance (local reads) have no problem hitting 300mb/sec. (also a bit low, but plenty enough for my purposes).

Thankful for any insight.
 

jwinsor566

n00b
Joined
Feb 26, 2011
Messages
13
ZFS caching (RAM or SSD) is ignoring large files.
Thats really very good but it will improve I/O only or access to small files or metadata like directory infomation, not sequential speed with large files.

to improve this, you need spindels, Ram, CPU and should avoid dedup, enryption and compression. For virtual machines, you can use VMware VMXNET3 virtual network driver. If you need real power, use SSD datadrives instead of harddisks.



correct

Gea, So would you say that my results are what you would expect with my current configuration? 50MB /s Write speeds? Or does this seem low to you? I have used VMXnet3 adapters in the windows and linux VM's but Nexenta does not support it yet so I cannot use that yet.

These were my results as I added spindles. I have not been able to get that 95 write speed however since I added the ZIL Not sure what else might have changed. I think I may have changed the "Write Back Cache" option to enabled or disabled to force zil use.

1 Mirror 2 Mirrors 3 mirrors 4 mirrors 4 Mirrors w/ SSD Zil
Seq 232 / 11 199.5 / 32 157 / 63 226 / 95 150 / 45
512k 217 / 26 203 / 35 204 / 44 201 / 72 153 / 58
4k 8 / 5.6 8 / 6.4 8 / 6.7 8 / 7.6 7.6 / 7
4k QD32 79 / 10 69 / 22 84 / 30 88 / 82 129 / 34
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
I am copying over vmxnet3 using Solaris 11 under ESXi right now (source = W7 over GbE) and am maxed out on a single path GbE - getting about 95MB/sec copying losslessly compressed HD video sustained. FWIW, as a data point. Source is W7 and a SSD. Target is a 4 disk raidz pool of Samsung 2TB F4s for testing, over regular Intel ICH10. ESXi is provisioning the disks to the VM as big ol' VMDKs.

Virtualcenter is showing some 6GHz consumed of my um... what is it... an i5 750 proc. However mpstat is only showing 30% used of two cores. I am only assigning 2 VCPU since i discovered that Solaris 11 doesn't seem to boot reliably with > 2 vCPU. Anyone notice this, btw?

Oh, NIC on the source is some Marvell junk. Source on the target is a $$$$ quad-port Intel ET2, but just using one port so far. This is over SMB, nothing special.

So far I am fairly impressed. I suppose I should do some in-host testing to see how good this really gets over the 10Gb adapter but the ESXi host (the i5 750 box I had lying around) only has 4GB RAM so I can't tossed up multiple VMs yet until i can get more RAM.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
About the answer "you need spindles", in my example, I have 16 spindles, of fairly new 2TB drives. We all know any of those disks would in windows manage ~100mb, and 2 striped together would almost certenly cap 125mb/sec (1Gbit connection) in simple large file reads.

My (our, I guess) question is, why does solaris/zfs based systems have so much trouble doing it with 16 spindles, again, on simple, sequential large-file reads ?

And, why, when non-networked performance (local reads) have no problem hitting 300mb/sec. (also a bit low, but plenty enough for my purposes).

Thankful for any insight.

All one can say, is

- SUN/ Oracle Solaris is one of the most stable and fast Enterprise operating systems.
- ZFS is one of the most sophisticated data management technologies with nearly unique data security


You have to say, despite the second, you can build really fast ZFS Systems that use all or some of the superior features like end to end checksums, ZFS Raid without Raid 5/6 write hole problems, ZFS Raid-Level 3, Copy on Write, encryption, compression, multiple copies among a lot of other features. My newest boxes with SSD pools are really fast with sequential rates > 1 Gb/s

Do not/ never look at the RAW sequential disk speed. Its only reached if you can stream a single large file by a single user. But that is not the usual ZFS use case. Mostly you have to look more at I/O power and transactional performance. Thats the reason why small 15k high speed disks have sometime the same or lower sequential transfer rates but a much much better end-result.

If you want to tune or build a ZFS system, you must look for weaknesses in a chain of hard- and software components
where the weakest part defines the result. Each config has its own and other problems and dependencies between the parts.

If you want to go for sure, look at commercial NexentaStor EE Boxes or Oracle systems and use similar parts
or always follow the best to use and well and often tested suggestions and avoid options that may cause problems.


Most problems rely on:

- not enough RAM
- Disks are slow and/or 4k
- CPU not fast enough
- Controller is slow or drivers are not optimal/ not well supported
- Nic is slow or not well suported (ex Realtex)
- not adequate ZFS pool design (number of disks, wrong optimazation speed vs capacity)
- not adequate use of options (ex dedup and cpmpression if you want best speed)
- bad cabling and Switch problems

- be realistic, do not expect a Porsche at no cost


Gea
 
Last edited:

DJ_Datte

Weaksauce
Joined
Oct 11, 2010
Messages
89
All one can say, is

- SUN/ Oracle Solaris is one of the most stable and fast Enterprise operating systems.
- ZFS is one of the most sophisticated data management technologies with nearly unique data security


You have to say, despite the second, you can build really fast ZFS Systems that use all or some of the superior features like end to end checksums, ZFS Raid without Raid 5/6 write hole problems, ZFS Raid-Level 3, Copy on Write, encryption, compression, multiple copies among a lot of other features. My newest boxes with SSD pools are really fast with sequential rates > 1 Gb/s

Do not/ never look at the RAW sequential disk speed. Its only reached if you can stream a single large file by a single user. But that is not the usual ZFS use case. Mostly you have to look more at I/O power and transactional performance. Thats the reason why small 15k high speed disks have sometime the same or lower sequential transfer rates but a much much better end-result.

If you want to tune or build a ZFS system, you must look for weaknesses in a chain of hard- and software components
where the weakest part defines the result. Each config has its own and other problems and dependencies between the parts.

If you want to go for sure, look at commercial NexentaStor EE Boxes or Oracle systems and use similar parts
or always follow the best to use and well and often tested suggestions and avoid options that may cause problems.


Most problems rely on:

- not enough RAM
- Disks are slow and/or 4k
- CPU not fast enough
- Controller is slow or drivers are not optimal/ not well supported
- Nic is slow or not well suported (ex Realtex)
- not adequate ZFS pool design (number of disks, wrong optimazation speed vs capacity)
- not adequate use of options (ex dedup and cpmpression if you want best speed)
- bad cabling and Switch problems

- be realistic, do not expect a Porsche at no cost


Gea

All you say is entirely correct. I work with enterprise storage in a large environment, but we don't have ZFS. I am toying with ZFS at home, but with quite hefty hardware.

Obviously, what you are answering to is not my question, but general performance advice. And I completely understand that. But it wasn't my question. My question was simply. What is holding it back in a rather simple, no heavy load scenario. There is quite obviously some bottleneck there.

You are quoting that you are getting 1GBps on you SSD pools. Is that over a network? what transport ? if not, it's not quite similar, as I am getting perfectly reasonable transfer figures too when avoiding the network bottleneck. so we can both agree, ZFS is fine.

But there is something there that chokes when involving network transfers, that does not happen on windows storage systems, and I am just simply curious what it is. The reports are repetitive on this board, and others. It's not just me. I am just curious to figure out what it is from people that have more experience then me. And I understand completely if it's a black hole, and not enough research has been done on the case. I am sure I will figure it out at some point, but if people have knowledge about it, it would be nice to know.
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
I will answer. My own initial testing has shown that Solaris network support and network stack efficiency may be a couple of steps behind other operating systems. I did not get good performance until I virtualized and put a $160 dual-port Intel card in there.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
I will answer. My own initial testing has shown that Solaris network support and network stack efficiency may be a couple of steps behind other operating systems. I did not get good performance until I virtualized and put a $160 dual-port Intel card in there.

You may ask at Oracle or Illumos forum. Only a OS developer can answer.

But for me thats a irrelevant question. Unless other systems have comparable storage-features,
i have no problem to use a faster machine for the same performance or to be restricted to
Intel based Nics or some LSI based storage controller.

Solaris is not mainstream. You cannot use what you have, you need to use whats best supported and
you must set priorities.

Gea
 
Joined
Dec 30, 2010
Messages
43
i can confirm that you cant setup ACL with win7 x64 ultimate in workgroup mode! users wont show up. i have installed win7 premium in virtualbox, logged in as root on my share and now all users appear when changing ACL.

can you please update the manual with this information. it can save many questions on the topic and hours of trying to get it work
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
i can confirm that you cant setup ACL with win7 x64 ultimate in workgroup mode! users wont show up. i have installed win7 premium in virtualbox, logged in as root on my share and now all users appear when changing ACL.

can you please update the manual with this information. it can save many questions on the topic and hours of trying to get it work


thanks for the info.
i will update the manual about this point

Gea
 
Joined
Dec 30, 2010
Messages
43
if i understand it right, it has something to do with workgroup mode and domains. premium doesnt support domains, other versions does
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,704
I wonder if this relates to the cifs doc on the nexenta website about win7 share access? There is some juju you need to do on your win7 box to make cifs stuff work right... I freely admit to knowing very little about this particular subject, so this might be a total red herring, in case it isn't though:

http://www.nexenta.com/corp/static/docs-stable/Win7CIFS.pdf
 
Last edited:

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
You may ask at Oracle or Illumos forum. Only a OS developer can answer.

But for me thats a irrelevant question. Unless other systems have comparable storage-features,
i have no problem to use a faster machine for the same performance or to be restricted to
Intel based Nics or some LSI based storage controller.

Solaris is not mainstream. You cannot use what you have, you need to use whats best supported and
you must set priorities.

Gea

It can be a problem for others, which is why it is important for people to ask the questions as well as to call out the limitations.

My suggestion, if network bandwidth is a critical factor, is to use Solaris 11 for its excellent vmxnet3 support + ESXi underneath for its highly optimized network stack with supported NICs. This way, you can rely on ESXi to shore up one of Solaris's long-standing weaknesses: x86 hardware support. See, to me, reliability is the most important qualifier, and I consider hardware support to be an important reliability feature. (If something breaks, I don't want to be at the mercy of 1 or 2 vendors supplying only 1 or 2 compatible replacement parts... neither do you.) I don't expect Solaris to match the broad hardware support that you get with Windows and increasingly Linux. I do notice that ESX is doing a better and better job, however, and that can give you the layer of abstraction you need to make "incompatible" hardware play nice with Solaris.
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
I couldn't get it running. Hence, Solaris Express 11, which supports it just fine. Just install the vmtools, plumb the interface, and call it a day.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,704
Hmmm, well, it isn't an issue for me now, but if&when I move to the all in one, I may need to move the pool to SE11, unless OI comes up with a vmxnet3 driver (assuming it doesn't have one now - I can take a look later tonight and check...)
 

Silenus

Limp Gawd
Joined
Jan 6, 2011
Messages
177
It's Nexenta that doesn't have vmxnet3 working. I have had it working on both OI and SE11. Not built in though. You have to load the VMWare tools then install the driver.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,704
Ah, okay, thanks. It's not important now, obviously. You said you couldn't get OI working?
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
That was me, and for all I know it was user error. I installed both and found SE11 smoother, however that could have been my particular hardware or some other issue. I decided I prefer SE11 more, and I may be able to get updates from Oracle which makes it a more natural choice for me.
 
Joined
Dec 30, 2010
Messages
43
has someone been successful to install or build transmission 2.22 torrent service? in the package manager is a very old version.
i did try to search with experimental package site
http://staticdev.uk.openindiana.org:10002/en/search.shtml?token=transmission&action=Search
when i click install and open with packagemanager nothing happens...

in freebsd i was using openvpn which worked perfectly. but in packagemanager openvpn is not found.. only option is to build from sourcefile?
 
Last edited:

Rhunt

n00b
Joined
Apr 28, 2011
Messages
9
I have been digging into trying to get email to work to the gmail server with no success but believe I'm getting closer.

We need Net::SMTP::TLS to send email via TLS for Gmail. I'm using OI Live so I did:

1) Install Napp-it as Gea has documented (when finished you will have set the root password which is required for the next step).

2) From System/Administration/Package Manager install Developement/Perl/net-ssleay

3) Open a terminal
> sudo perl -MCPAN -e shell ! 'no' for automatic config
>> install Net::SMTP::TLS ! accept all dependencies
>> exit
> exit

At this point I think you will have functional Net::SMTP::TLS.

I've attempted to hack the test smtp script but likely missed something or it just will not work. Here is some documentation: http://search.cpan.org/~awestholm/Net-SMTP-TLS-0.12/lib/Net/SMTP/TLS.pm

The last part is to identify where the email script is used in Gea's Napp-it code. All places where standard smtp is used need to be replaced by the tls version.

I'll continue to try find the solution but I'm not a perl programmer....

Thx, Rod
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
I have been digging into trying to get email to work to the gmail server with no success but believe I'm getting closer.

We need Net::SMTP::TLS to send email via TLS for Gmail. I'm using OI Live so I did:

1) Install Napp-it as Gea has documented (when finished you will have set the root password which is required for the next step).

2) From System/Administration/Package Manager install Developement/Perl/net-ssleay

3) Open a terminal
> sudo perl -MCPAN -e shell ! 'no' for automatic config
>> install Net::SMTP::TLS ! accept all dependencies
>> exit
> exit

At this point I think you will have functional Net::SMTP::TLS.

I've attempted to hack the test smtp script but likely missed something or it just will not work. Here is some documentation: http://search.cpan.org/~awestholm/Net-SMTP-TLS-0.12/lib/Net/SMTP/TLS.pm

The last part is to identify where the email script is used in Gea's Napp-it code. All places where standard smtp is used need to be replaced by the tls version.

I'll continue to try find the solution but I'm not a perl programmer....

Thx, Rod

you just need to create a new menu item and insert the code from the example:

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS(
'your.mail.host',
Hello => 'some.host.name',
Port => 25, #redundant
User => 'emailguy',
Password=> 's3cr3t');
$mailer->mail('emailguy@your.mail.host');
$mailer->to('someonecool@somewhere.else');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;


then you have to replace the dummies with your real data like

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS(
'mail.google.de',
Hello => 'your.host.name',
Port => 25, #redundant
User => 'your user name',
Password=> 'your pw');
$mailer->mail('yourusername@mail.google.de');
$mailer->to('someol@somewhere.else');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;

and save/start to try


Gea
 

Rhunt

n00b
Joined
Apr 28, 2011
Messages
9
you just need to create a new menu item and insert the code from the example:

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS(
'your.mail.host',
Hello => 'some.host.name',
Port => 25, #redundant
User => 'emailguy',
Password=> 's3cr3t');
$mailer->mail('emailguy@your.mail.host');
$mailer->to('someonecool@somewhere.else');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;


then you have to replace the dummies with your real data like

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS(
'mail.google.de',
Hello => 'your.host.name',
Port => 25, #redundant
User => 'your user name',
Password=> 'your pw');
$mailer->mail('yourusername@mail.google.de');
$mailer->to('someol@somewhere.else');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;

and save/start to try


Gea

Thanks Gea, that was the trick! With TLS installed I have success with the following:

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS('smtp.gmail.com',
Hello => 'smtp.gmail.com',
Port => 587, #redundant
User => 'your.id@gmail.com',
Password=> 'your.pw');
$mailer->mail('your.id@gmail.com');
$mailer->to('target@email.com');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;

Now

1) How do I identify where the email notifications are embedded in the napp-it scripts?

2) Will I be able to retain compatibility to future napp-it releases?

Thanks again for your help.

Rod
 

barbz

Weaksauce
Joined
Apr 25, 2010
Messages
85
has someone been successful to install or build transmission 2.22 torrent service? in the package manager is a very old version.
i did try to search with experimental package site
http://staticdev.uk.openindiana.org:10002/en/search.shtml?token=transmission&action=Search
when i click install and open with packagemanager nothing happens...

in freebsd i was using openvpn which worked perfectly. but in packagemanager openvpn is not found.. only option is to build from sourcefile?

Im also trying to figure out how to do this. I have tried adding in the staticdev repo but I still see 1.93 not 2.22

this is the one thing holding me back from moving onto oi/sol11 (lack of 2.xx transmission or another daemon substitute).

Paul

Edit: added in a the package tree at http://staticdev.uk.openindiana.org and installed sfetransmission and got 2.22 , then disabled that package tree.

Not sure how safe this is considered to be.

Paul
 
Last edited:

rootdown

n00b
Joined
Jan 6, 2005
Messages
22
That was me, and for all I know it was user error. I installed both and found SE11 smoother, however that could have been my particular hardware or some other issue. I decided I prefer SE11 more, and I may be able to get updates from Oracle which makes it a more natural choice for me.

I'm sorry if this is a dumb question (and i've been reading this thread for a while) but, there won't be public updates of SE11 in the future?
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
It's unlikely. Oracle's model is well-established with Oracle Enterprise Linux and it looks like Solaris 11 and Express 11 are following the same model. Specifically, OS is free, but updates require a support contract. If they follow their approach to OEL, then we can expect to see occasional updated Solaris 11 builds (with the latest patches available at release time) show up.
 

sphinx99

Gawd
Joined
Dec 23, 2006
Messages
945
I've been unable to get OI or SE11 to boot reliably under ESXi with more than 2 vCPU allocated to it. 2 vCPU is absolutely reliable. 3 vCPU = boots maybe half the time, other half of the time it freezes at the initial two-line OS banner. 4 vCPU = pretty much always hangs at the banner.

Is it just me or have others experienced this?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
I've been unable to get OI or SE11 to boot reliably under ESXi with more than 2 vCPU allocated to it. 2 vCPU is absolutely reliable. 3 vCPU = boots maybe half the time, other half of the time it freezes at the initial two-line OS banner. 4 vCPU = pretty much always hangs at the banner.

Is it just me or have others experienced this?

this is a known problem.
use 2 vCPU. Its also not as critical as having enough RAM.

From german VMware forum, i was told to use always no more than 2 vCPU for guests.


Gea
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
Thanks Gea, that was the trick! With TLS installed I have success with the following:

use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS('smtp.gmail.com',
Hello => 'smtp.gmail.com',
Port => 587, #redundant
User => 'your.id@gmail.com',
Password=> 'your.pw');
$mailer->mail('your.id@gmail.com');
$mailer->to('target@email.com');
$mailer->data;
$mailer->datasend("Sent thru TLS!");
$mailer->dataend;
$mailer->quit;

Now

1) How do I identify where the email notifications are embedded in the napp-it scripts?

2) Will I be able to retain compatibility to future napp-it releases?

Thanks again for your help.

Rod

you may modify a already created autojob (in _log/jobs: jobid.pl).
Such a modification is independent from updates.

I will try to add the module in one of next updates and give an option in settings to use it.
(hope its a copy and run capable module without a lot of dependencies)

ps
i have updated
http://napp-it.org/napp-it/all-in-one/index.html with this item


Gea
 
Last edited:

Rhunt

n00b
Joined
Apr 28, 2011
Messages
9
you may modify a already created autojob (in _log/jobs: jobid.pl).
Such a modification is independent from updates.

I will try to add the module in one of next updates and give an option in settings to use it.
(hope its a copy and run capable module without a lot of dependencies)

ps
i have updated
http://napp-it.org/napp-it/all-in-one/index.html with this item


Gea

Thanks again Gea.

Learning as fast as I can....

Rod
 
Joined
Dec 30, 2010
Messages
43
Gea, can you add total system uptime to napp-it start page? in freebsd i had spontaneous reboots. in solaris i didnt happen past 5 days. would be usefull to login to nappit and check uptime and pool status in 1 page :)

do you have some kind of roadmap for napp-it? like new features/services?
 
Last edited:
Top