News : The level of daily SPAM has reached insane proportions, all registrations are now manual. I ask you to send me an e-mail (john (at) murga (dot) org), to confirm that you want me to create an account for you.


Post Reply  Post Thread 
Pages (3): « First < Previous 1 [2] 3 Next > Last »
murgaLua CGI %ENV
Author Message
Juergen
Member
***


Posts: 81
Group: Registered
Joined: May 2007
Status: Offline
Reputation: 0
Post: #11
RE: murgaLua CGI %ENV

JohnMurga Wrote:
Not really a fair comparison as once you incorporate the relevant bindings to stock lua the startup time will increase too... Although if you are going to use murgaLua for CGI you probably should unpack it.

Would be interesting to measure it. But I guess it will still be a lot faster even when the sql bindings are loaded dynamically.

JohnMurga Wrote:
FYI, the XML bindings and a few other bits and pieces are also not available elsewhere ;-)

I know, but most functionality is available in another form on luaforge.

JohnMurga Wrote:
BTW, I'd like to see a test to prove your 30% file overhead claim from an earlier post.


Code:
unction iotest(filename, isize)
  local obytes=0
  local video=assert(io.open(filename,"r"),"Cant open file: "..filename)
  local devnull=assert(io.open("/dev/null","w"),"Cant open /dev/null")
  local size=video:seek("end")
  video:seek("set")

  local vs,pos,iterations
  iterations=size/isize

  while(iterations>=0) do
    vs=video:read(isize)
    devnull:write(vs)
    obytes=obytes+#vs
    iterations=iterations-1
  end
  video:close()
  return obytes
end
if type(arg[1])~="string" then print("You must prowide a filename as the first argument!") os.exit(1) else fn=arg[1]  end
if not arg[2] or not tonumber(arg[2]) then print("You must prvide a chunksize as the second argument!") os.exit(1) else cs=tonumber(arg[2]) end
print(iotest(fn,cs).." bytes written to /dev/null")


You should save that as io_speed.lua

and this:

Code:
#!/bin/bash
export TIMEFORMAT="%5R %5U %5S"
blocksize=${2:-10000}
filename=${1:?"The first parameter should be a large file! (additionally blocksize and runs could also be specified)"}
if ! [ -f $filename ] ; then echo "the file should exist!" ; exit -1 ; fi
runs=${3:-20}
offset=3 #to make sure the caches are warm
stock_lua=lua5.1
murgaLua=murgaLua
luascript=io_speed.lua
if ! [ -f $luascript ] ; then "where is the lua script?" ; exit -1 ; fi

function io_test () {
   local interp="$1"
   local pref="$2"
   shift 2
   local param="$@"
   for ((i=1;i<=runs;i++)) do
     echo run: $i with $interp $param
     tstring=$({ time $interp $param > /dev/null 2>&1 ; } 2>&1)
     set -- $(IFS=' ' ; echo $tstring)
     echo real: $1 user: $2 system: $3
     eval $pref\_real[$i]=$1
     eval $pref\_user[$i]=$2
     eval $pref\_system[$i]=$3
   done
}

function calculate_result() {
   local pref=$1
   result_real=0; result_system=0; result_user=0
   local real=$pref"_real" user=$pref"_user" system=$pref"_system"
   for ((i=$offset;i<=runs;i++)) do
     result_real=$(eval dc -e \"5 k $result_real \${$real[$i]} + p\")
     result_user=$(eval dc -e \"5 k $result_user \${$user[$i]} + p\")
     result_system=$(eval dc -e \"5 k $result_system \${$system[$i]} + p\")
   done
   result_real=$(eval dc -e \"5 k $result_real $runs $offset -  / p\")
   result_user=$(eval dc -e \"5 k $result_user $runs $offset -  / p\")
   result_system=$(eval dc -e \"5 k $result_system $runs $offset - / p\")
   eval $pref\_real[0]=$result_real
   eval $pref\_user[0]=$result_user
   eval $pref\_system[0]=$result_system
   echo real: $result_real user: $result_user system: $result_system
}
  
io_test $stock_lua "slo" "-e os.exit(0)"
calculate_result "slo"
io_test $murgaLua "mlo" "-e os.exit(0)"
calculate_result "mlo"
io_test $stock_lua "sl" $luascript $filename $blocksize
calculate_result "sl"
io_test $murgaLua "ml" $luascript $filename $blocksize
calculate_result "ml"
echo startup overhead: $(eval dc -e \"5 k ${mlo_real[0]} ${slo_real[0]} - p\")"s" which is a $(eval dc -e \"5 k ${mlo_real[0]} ${slo_real[0]} / p\") times overhead
echo average file copy time for stock lua is: ${sl_real[0]} and for murgaLua: ${ml_real[0]} which is a $(eval dc -e \"5 k ${ml_real[0]} ${mlo_real[0]} - ${sl_real[0]} ${slo_real[0]} - / 1 - 100 *p\") "% overhead"


as what you want.

The io_speed.lua is from an earlier experiment and copies a file to /dev/null (so it is Unix specific).

I also wrote a small bash script to test it out, because I found a kernel bug on my system when I tested it today. It should work on any linux system (if dc is installed).

Juergen

03-24-2008 06:57 AM
Find all posts by this user Quote this message in a reply
JohnMurga
Administrator
*******


Posts: 381
Group: Administrators
Joined: Apr 2007
Status: Offline
Reputation: 2
Post: #12
RE: murgaLua CGI %ENV

Juergen Wrote:
Would be interesting to measure it. But I guess it will still be a lot faster even when the sql bindings are loaded dynamically.

I am not entirely sure ...

Juergen Wrote:

JohnMurga Wrote:
FYI, the XML bindings and a few other bits and pieces are also not available elsewhere ;-)

I know, but most functionality is available in another form on luaforge.

Things that work and I have chosen to incorporate, yes ...
But most of the XML api is home grown as I didn't find something that did what I wanted.

Juergen Wrote:

JohnMurga Wrote:
BTW, I'd like to see a test to prove your 30% file overhead claim from an earlier post.

You should save that as io_speed.lua

Thanks !

That'll allow me to fine tune the build options and hopefully improve things.

Cheers
JohnM

03-24-2008 07:11 AM
Visit this user's website Find all posts by this user Quote this message in a reply
Juergen
Member
***


Posts: 81
Group: Registered
Joined: May 2007
Status: Offline
Reputation: 0
Post: #13
RE: murgaLua CGI %ENV

JohnMurga Wrote:

Juergen Wrote:
Would be interesting to measure it. But I guess it will still be a lot faster even when the sql bindings are loaded dynamically.

I am not entirely sure ...

Actually 1ms is on a now common machine with 2GHz is about 2million CPU cycles. That is an awful lot of cycles to spend for a few thousand relocations. The Linux dynamic loader isn't that slow (actually it is quite fast). I did a short test and compiled a lsqulite3.so with the newest libsqlite (3.5.7) compiled in statically.
With the following test:

Code:
require"lsqlite3"
db = sqlite3.open('test.db')
os.exit(0)

It takes about 6ms to complete (an empty test.db exists afterward and I checked that indeed the library has been loaded successfully). Compared with the 4ms it takes to complete "lua -e "os.exit(0)" this is a 2ms overhead. (Of course everything with hot caches).
So main factor of the huge startup difference is UPX. On the machine I'm currently sitting, it takes about 210ms to start murgaLua packed compared with 40ms unpacked. But there is still a difference compared with 6ms.

JohnMurga Wrote:
Things that work and I have chosen to incorporate, yes ...
But most of the XML api is home grown as I didn't find something that did what I wanted.

There is also Michal Sweets mxml library which is also small and fast and it shouldn't be hard to write a binding if not already done.

JohnMurga Wrote:
That'll allow me to fine tune the build options and hopefully improve things.

Do you want to use selective optimization levels for some files?

Juergen

03-25-2008 05:55 AM
Find all posts by this user Quote this message in a reply
mikshaw
Senior Member
****


Posts: 522
Group: Registered
Joined: Apr 2007
Status: Offline
Reputation: 5
Post: #14
RE: murgaLua CGI %ENV

I have a belief that a person can be too picky about web server performance if that server is on a personal machine and does not have much traffic. When you consider the fact that most people's machines are typically loaded with a lot of waste simply to run a desktop, and those same people often run their servers on a graphical system, the overhead of the gui crap they run greatly outweighs the performance difference between Lua and murgaLua.

I run a local server using murgaLua (still compressed) and its response is instant. If I were to run that same server on a ubuntu or windows system I wouldn't be surprised if Lua proper actually had lower performance than murgaLua does on my DSL/Slackware box.
That's just speculation, of course, but it's just to express my belief that a lot of people seem to put extra emphasis on slim applications while at the same time they are running fat operating systems.

04-12-2008 12:51 PM
Find all posts by this user Quote this message in a reply
JohnMurga
Administrator
*******


Posts: 381
Group: Administrators
Joined: Apr 2007
Status: Offline
Reputation: 2
Post: #15
RE: murgaLua CGI %ENV

There are many issues with murgaLua ...

The startup time is too high, and it is too big right now ...

Much of the startup time is murgaLua and LuaSocket initialization, however, I believe that for 0.7.X I can improve that dramatically (I have done a few tests), and I also think I can even get murgaLua small again ... Either way I had a self imposed ban on any new bindings after 0.6.X, so I guess I have gone a little crazy with that recently.

PS ... I have changed the build options for 0.6.8

Cheers
JohnM

04-14-2008 03:36 AM
Visit this user's website Find all posts by this user Quote this message in a reply
Juergen
Member
***


Posts: 81
Group: Registered
Joined: May 2007
Status: Offline
Reputation: 0
Post: #16
RE: murgaLua CGI %ENV

JohnMurga Wrote:
There are many issues with murgaLua ...

The startup time is too high, and it is too big right now ...

Of course, this depends on personal perspective, but the startup time is really high. For the size, it shouldn't be a problem on normal users.
It would also make sense to document somewhere that it makes really sense to uncompress murgaLua, when the binary size doesn't matter.

JohnMurga Wrote:
Much of the startup time is murgaLua and LuaSocket initialization, however, I believe that for 0.7.X I can improve that dramatically (I have done a few tests), and I also think I can even get murgaLua small again ... Either way I had a self imposed ban on any new bindings after 0.6.X, so I guess I have gone a little crazy with that recently.

I think what would really make sense (although it is a little bit late) to do the initialization only when the functionality is required. (in this case the user has to use require() to initialize it)

JohnMurga Wrote:
PS ... I have changed the build options for 0.6.8

Yes, you exported the symbols (although I'm not sure if you got the order right (putting the -Wl,-E after the import of the static lua library). On the other hand this doesn't matter in this case at all, since you stripped them away anyhow. ;-)

Juergen

This post was last modified: 04-15-2008 01:33 AM by Juergen.

04-15-2008 01:30 AM
Find all posts by this user Quote this message in a reply
Juergen
Member
***


Posts: 81
Group: Registered
Joined: May 2007
Status: Offline
Reputation: 0
Post: #17
RE: murgaLua CGI %ENV

mikshaw Wrote:
I have a belief that a person can be too picky about web server performance if that server is on a personal machine and does not have much traffic. When you consider the fact that most people's machines are typically loaded with a lot of waste simply to run a desktop, and those same people often run their servers on a graphical system, the overhead of the gui crap they run greatly outweighs the performance difference between Lua and murgaLua.

I run a local server using murgaLua (still compressed) and its response is instant. If I were to run that same server on a ubuntu or windows system I wouldn't be surprised if Lua proper actually had lower performance than murgaLua does on my DSL/Slackware box.
That's just speculation, of course, but it's just to express my belief that a lot of people seem to put extra emphasis on slim applications while at the same time they are running fat operating systems.

Don't overestimate the desktop overhead. The biggest overhead is memory wise. But if everything is done right (there is still a lot of software out there that does ridiculus things, like waking up every few 100 us to check someting which could be better done using the right mechanisms), there shouldn't be that much overhead, except for the memory usage. The most overhead on the desktop, which I experience on my machines, is when I browse and hit a web site which shows a dozen or more flash animations. Unfortunately that can even max out a fast machine.

And yes, if you write something with a web GUI that spawns off a murgaLua CGI once in a while, you will hardly observe a speed problem. But if you use it for a site which is used by a few users concurrently and some web pages call multible CGI scripts, 100ms startup overhead leads pretty fast to big trouble.

On the other hand, while UPX packing makes murgaLua much smaller (and hides the real size), this benefit only matters when you have very limited space on your storage medium. In any other case it is very counter productive, because it will cost you a lot of memory if multiple copies of murgaLua run concurrently, because page sharing and demand paging obviously doesn't work with UPX compressed binaries.
So if you have say 5 instances of murgaLua running, the difference is (roughly estimated for the dynamic binary) more than 7Mb in memory consumption.

Juergen

04-15-2008 02:27 AM
Find all posts by this user Quote this message in a reply
JohnMurga
Administrator
*******


Posts: 381
Group: Administrators
Joined: Apr 2007
Status: Offline
Reputation: 2
Post: #18
RE: murgaLua CGI %ENV

Juergen Wrote:
So if you have say 5 instances of murgaLua running, the difference is (roughly estimated for the dynamic binary) more than 7Mb in memory consumption.

Not that most people would notice 7Mb, however, the way to do it right would be to have a murgaLua server that can service those 5 concurrent users in separate threads. Having a thread pool and a compiled GCI lua cache then would get rid of most of the overhead and give you a very optimized experience ... I believe that writing a basic web server along these lines won't be hard, so I'll give it a go.

Anyway ...

My focus from now on is performance, size and more FLTK bug fixes and features.

Juergen, I could really do with your feedback on the luasys functionality and some of the other stuff in 0.6.8 ...

Cheers
JohnM

04-15-2008 04:17 AM
Visit this user's website Find all posts by this user Quote this message in a reply
JohnMurga
Administrator
*******


Posts: 381
Group: Administrators
Joined: Apr 2007
Status: Offline
Reputation: 2
Post: #19
RE: murgaLua CGI %ENV

Juergen Wrote:
Yes, you exported the symbols (although I'm not sure if you got the order right (putting the -Wl,-E after the import of the static lua library). On the other hand this doesn't matter in this case at all, since you stripped them away anyhow. ;-)

That is not what I was referring to ...

If you try third party modules with the new executable you may find it no longer segfaults on you.

Cheers
JohnM

04-15-2008 07:28 AM
Visit this user's website Find all posts by this user Quote this message in a reply
dvw86
Junior Member
**


Posts: 39
Group: Registered
Joined: Apr 2007
Status: Offline
Reputation: 0
Post: #20
RE: murgaLua CGI %ENV

It's been a long time but I have still been playing with this. I have a web based application that is using murgaLua. My wife has been using it frequently for her volunteer work for about a year now, and it has done very well. I tried just using Lua and adding the in the needed modules, but I found that I was basically building murgaLua. It is designed to run on a stand alone machine and only listen to the loopback. Doing it this way, performance has been great. In Windows XP and Windows 7 I am using the Mongoose web server. In Mac OSX and Ubuntu I am using Apache. Both work very well with no difference to the user. I am now starting to get some interest in my application at a corporate level and I was hoping that interest in murgaLua hasn't died. I'm sure that a lot has gone on in everyone's lives since I was last on here. I hope all is well.

05-20-2012 02:45 AM
Find all posts by this user Quote this message in a reply
Post Reply  Post Thread 

View a Printable Version
Send this Thread to a Friend
Subscribe to this Thread | Add Thread to Favorites

Forum Jump: