Do you think ScummVM release testing is cut too short?

[joachimeberhard]

I am getting feedback from many friends who are MacOSX users who are not able to start ScummVM 0.8.0 at all.

While I can not verify them all right now, it still is what I just had in my blood regarding the 0.8.0 release.

I am visiting my neighbor in half an hour, he has OSX and also claims, that 0.8.0 doesn't work at all. Even though I'm no Mac specialist, I'm trying to find out what is wrong there and wether this is true.

Best regards

Joachim

[sev]

It was found out here on forums and a bugreport submitted, that our build requires recent MacOS X to run. Probably we need to loose that requirement. But I'm not MacOS user.

And another note. If you'll look at our history, we always released 0.x.1 version which had bugfixes. The nature of users is such, that we received bugreports on 0.7.1 and even 0.6.1 during test period. They just didn't care to test it well. It is not only with our project but with most of them.

And in any case, complaining now will bring no good, just will upset us, developers more. This release was delayed too long anyway.


Eugene[/u]

[joachimeberhard]

And in any case, complaining now will bring no good, just will upset us, developers more. This release was delayed too long anyway.
Eugene[/u]

I feel sorry Eugene that this came up all wrong.

I didn't want to upset you.

I just wanted to tell you my impression from a heavy tester's point of view.

But as you have already said, maybe seen 0.8.0 as an extended beta (as the readme clearly states ScummVM should be considered beta), the 0.8.0 release was indeed something to put out now. Moreso when considered, that users still file bugreports for 0.6.0 or so.

Considered the possibility of a bugfix release within months, I personally understand your plans from a developers point of view.

I really did not mean to tell you to wait another 4 months until release, but I thought another 1-2 weeks, maybe even something as Simple as ScummVM0.8.0 BETA program (some people just don't understand what CVS means or get stuck because they think they have to compile something) could help.

Also, I really think you should state parts of the release plans on the main site. Or discuss some thoughts with the users here. I mean, if you openly state release-plans on the scummvm-devel, you could also state it on the website.

There are really some good playtesters out there, who just aren't good in technical stuff.

Please, sev, I don't mean to be rude to the developers in any way.

In fact, I'm really a fan of you guys!

But I just try to give you an impression from a testers point of view.

Keep up the good work & thank you all for the greatest project of the world:

ScummVM!

Joachim

[fingolfin]

I have to respond to some specific complaints named here. And folks, please be careful in what you write, and do not make hasty assumptions about causes and reasons of issues when you have little to no idea about what you are talking. It merely annoys people like me who spend a lot of their spare time to work on ScummVM, only to be told afterwards how stupid they are for doing things the way they do them.

Well thought constructive criticism is always welcome, though, as are inqueries about the reasons for a problem.

Now, to respons specifically to some statements made here.

However, I still believe that 0.8.0 was rushed out the door. Using the Mac OS X build as example you stated that there are three areas where the Mac codebase differs the standard codebase. They are endianness, AltiVec and audio drivers.

You already mentioned that sword2 alignment problem (endianness) so, that is one out of three having an issue.

This is incorrect, as eriktorbjorn already pointed out. Alignment is not endianess, and we are not aware of any Sword2 problems on Mac OS X. If you are, please file a report.

Looking at the bug tracker shows priority 6 bug [ 1342732 ] MAC: hq2x/hq3x filters broken when altivec is enabled in which fingolfin had to disable the AltiVec code in 0.8.0 in order for the hq filters to work.

This is not a 0.8.0 release issue. It's build issue triggered by a compiler bug. It may have been caught had we publish a testing build for Mac OS X. That is a valid criticism, and we'll keep that in mind for the next release cycle. However it is just plain wrong to attribute this to a "rushed release". Even with a 3 months testing period it may not have been caught -- the problem here was simply that apparently nobody tested BS2 + HQx + Altivec recently (since the switch to GCC 4.0 by me, that is).

That's two out of three having an issue. Further up the bug tracker shows two possibly related priority 5 issues (correct me if I'm wrong) [ 1344631 ] build fails on Mac OS X 10.2.8 when MT32 emulation is used and [ 1343971 ] MAC: ScummVM 0.8.0 fails to launch which might be related to a conflict between Mac OS X 10.2.8 and MT32 emulation through the audio drivers.

Neither of the two issues is related to the audio drivers! Not in the least.

The first one is a build problem on an old OS X version which we do not really support anymore, and is probably there since we added the MT32 code. Nobody (!) has reported the issue since then. Blaming this on 0.8.0 is silly.

The second one is again an issue with the OS X binary and in no way 0.8.0 specific. It's not nice, and we'll fix it, no worries.

No one is assigned to either of these cases so, there is no telling if they are duplicatable or related or if they actually is related to the audio drivers so, this one is just a potential at this point.

I take personal offense in this statement. It implies we are lazy since nobody has yet been assigned to these cases. As I stated above, and as you stated yourself, we are all volunteers. It can take us a couple of days before we get around to looking at an issue. We take all bug reports seriously, but we aren't a 24-7 paid support hotline, so please bear with us, for god's sake!

So definitively two of the three Mac specific areas have issues in the 0.8.0 release. Depending on the outcome 1344631 and 1343971 there may be an issue in the audio drivers as well making it three for three. Had there been at least one official CVS build for Mac OS X after the Oct 11 call for testers I truly believe that these would have been caught and corrected before 0.8.0 release. So, in my opinion an official CVS build for Mac OS X is not redundant.

Yes, you are right. However, due to limited resources, we can't provide regular CVS build for OS X. Next time around, though, we'll provide a couple builds, at least on request, via our web site.

[Kaminari]

maybe even something as Simple as ScummVM0.8.0 BETA

Hopefully I'm not going to sound harsh nor sarcastic, but if you read again the version number (0.8), you'll realize that there's a leading zero in it that officially suggests the program has been and will be a BETA program for quite some time. Any mention like "0.8 BETA" would sound naively redundant.

[joachimeberhard]

maybe even something as Simple as ScummVM0.8.0 BETA

Hopefully I'm not going to sound harsh nor sarcastic, but if you read again the version number (0., you'll realize that there's a leading zero in it that officially suggests the program has been and will be a BETA program for quite some time. Any mention like "0.8 BETA" would sound naively redundant.

As I've stated before, I know lots of people who do not understand that.
It was just a suggestion to cope with their confusion.

Calling it Beta is just a suggestion to actively encourage people in bugtesting before release.
But I think it's not too good if only 1 person has tested a game, and that there are too little volunteers (testusers) also.

With BETA i mean something like one version out on the webpage, prominently downloadable (Big red <Download here> Hehe.) and not changing code for 1 week before release or so. But this is only a suggestion, not that I think this would be the only smart way or knew it better.

So hopefully, no-one is sounding sarcastic.

But the proplem is, sometimes sarcasm can also be read, when there's none to read.
It often depends on the readers point of view.

Hopefully, I didn't make the developers sad with my posts.
I really love ScummVM, and I really appreciate your work!

But even though we are no programmers, we do have some points sometimes. And some of us do spend their sparetime too on this project, because we love it.

Best regards

Joachim

[sev]

With BETA i mean something like one version out on the webpage, prominently downloadable (Big red <Download here> Hehe.) and not changing code for 1 week before release or so. But this is only a suggestion, not that I think this would be the only smart way or knew it better.

Hmmm. I don't understand. Let's consider, next day we publish that binary, users find couple of serious bugs there and we fix them. What to do? Fix them, or they'll go to release? Let's say, we'd fix them and update the binary. Should we wait for another week? This approach never worked even with commercial software.

As of 0.8.1. If it will be released, that will happen approximately a month from 0.8.0 release, as we do that usually.


Eugene

[joachimeberhard]

Hmmm. I don't understand. Let's consider, next day we publish that binary, users find couple of serious bugs there and we fix them. What to do? Fix them, or they'll go to release? Let's say, we'd fix them and update the binary. Should we wait for another week? This approach never worked even with commercial software.

As of 0.8.1. If it will be released, that will happen approximately a month from 0.8.0 release, as we do that usually.


Eugene

Ok, this certainly is something different.

I really didn't know that if a bugfixrelease is comming, it is planned rather soon (compared to 0.7.1->8.0.0).

Somehow its really difficult to really understand what you are planning, when you only rely on scummvm-devel, as I do.

And from a users point of view, I want to get as much information as possible about ScummVM, without being nosey. This certainly makes it easier for us too.

But as we all can see, this forum is starting to be a convenient way of discussing, even for non-technical-people compared to SF.

So maybe it helps picky users like me to get a deeper understanding of the question: "What IS the Secret of ScummVM??"



And also, I think you can be quite proud of yourselves:

Most users voted, you did it right! So could there be any better feedback?

Joachim

P.S.: I really liked the Christmas release idea from last year

[Lord Savage]

Thank you fingolfin and eriktorbjorn for correcting my misconception about the alignment issue.

My intention from the beginning was to offer constructive criticism on this release, neither of my posts were written with malice in mind. For any offence that they may have caused, I apologize.

The main point of my second post was to try to convince the powers that be (The ScummVM development team) that there is still a need for the occasional CVS build for Mac OS X during the final testing phase. The following quote was simply meant to convey that the bug tracker does not have enough info in it to draw any concrete conclusions on these two issues. It was never intended to mean that I felt that the development team was lazy or did not care about the bugs listed. Nothing could be farther from the truth.

No one is assigned to either of these cases so, there is no telling if they are duplicatable or related or if they actually is related to the audio drivers so, this one is just a potential at this point.

I take personal offense in this statement. It implies we are lazy since nobody has yet been assigned to these cases. As I stated above, and as you stated yourself, we are all volunteers. It can take us a couple of days before we get around to looking at an issue. We take all bug reports seriously, but we aren't a 24-7 paid support hotline, so please bear with us, for god's sake!

Thank you once again for all your hard work.

[Kirben]

There was no official Macintosh snapshot of ScummVM provided for testing, but if you had checked the forums you would have noticed someone provided a Macintosh snapshot of ScummVM just before testing started. Specifically at http://forums.scummvm.org/archives.php?thread=1364367

[LogicDeLuxe]

considering that at least in the case of MI/MI2/Indy4 they don't seem to offer any advantage over the regular (and common) PC versions

Monkey Island 2 does.
There are a lot more sampled sound effects. The PC versios has only a few like spitting, screaming and a burb. Most others are cheap sounding Adlib, and even the MT-32 effects aren't that great. The FM-Towns version has sampled sounds for almost anything, like the bats, the fire, the hammer, the saw, the bell, the thunder etc.
Fortunately, that part seems to work fine with ScummVM.