From m.cave-ayland at webbased.co.uk Tue May 4 05:45:09 2004 From: m.cave-ayland at webbased.co.uk (Mark Cave-Ayland) Date: Tue, 4 May 2004 13:45:09 +0100 Subject: [postgis-devel] LWGEOM -- inital version ready for testing In-Reply-To: <8F4A22E017460A458DB7BBAB65CA6AE52188E6@openmanage> Message-ID: <8F4A22E017460A458DB7BBAB65CA6AE50265BA@openmanage> Hi Dave, Firstly a big well done for the work you've put into the LWGEOM - I had hoped to have taken a more active role in developing the code but unfortunately my time has been needed on other things :) Anyway, haven't seen many responses from the other guys on the list so thought I would add in my initial thoughts after looking at the CVS snapshot (I've yet to spend some time loading it into a database so this is mostly just thoughts on screen). 1. I like the reformat of the code. I know that some of the functions are missing, but the whole things seems easier to read, particularly in the use of function names etc. 2. At this stage it think it would be useful to include some sort of debugging function to dump the LWGEOM format into a human-readable form, for example: > SELECT raw_lwgeom(geom) FROM geomtable; | size | has_srid | has_bbox | dimensionality | wkb_type | wkt_geom | wkb_geom | +------+----------+----------+----------------+----------+----------+--- -------+ I can see this being useful for helping users in situations where they are not sure exactly how they have configured the particular geometry in the table. What do you think? 3. Should we prevent users from adding bounding boxes to point columns? (i.e. is the single/double precision conversion fast enough to make this a waste of disk space?) 4. I think it would be useful to include BBOXes in LWGEOM by default. My thinking here would be that those users who would be knowlegable enough to be concerned about the space saving will more than likely to be knowledgable enough to remove it where as new users may get confused when certain queries that used to perform well in PostGIS perform poorly using LWGEOM. 5. As far as I can see, assuming a non-index scan, the LWGEOM operators call the box2d_* functions directly which is defined using float4s. It looks like this is contrary to the OGC spec since all coordinates (and therefore I would guess operators) are defined as doubles? :(. I guess that we would need to maintain a box2d type which uses doubles as well as floats and use this for all the LWGEOM operators/functions (the box2d float4 would still be used for the indexes). Here it would be compulsory to add RECHECK to the operator classes since when expanding the box2d(double) to box2d(float) extra geometries may be returned by an overlap calculation. The RECHECK would ensure that these would be stripped out before the result set was returned. 6. It looks like you can add bboxes and srids to a geometry but not remove them? Also if I follow the instructions in the README to add bounding boxes to a column: DROP INDEX ; UPDATE SET = AddBBOX(); CREATE INDEX ON
USING GIST ( GIST_LWGEOM_OPS); VACUUM ANALYSE
; That would seem to work great. My question is now if I add more geometries to the table, would these new geometries not have a bounding box attached? Would the only way to add bounding boxes to these geometries be by running the update query and reindexing again? Anyway glad to see that this stuff is becoming a reality, and I hope that this will get some discussion started on the list about your work so far :) If I get a chance I will try and setup a test database with some sample data and see how it performs.... Cheers, Mark. --- Mark Cave-Ayland Webbased Ltd. Tamar Science Park Derriford Plymouth PL6 8BX England Tel: +44 (0)1752 764445 Fax: +44 (0)1752 764446 This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. > -----Original Message----- > From: postgis-devel-bounces at postgis.refractions.net > [mailto:postgis-devel-bounces at postgis.refractions.net] On > Behalf Of David Blasby > Sent: 29 April 2004 19:52 > To: postgis-devel at postgis.refractions.net > Subject: [postgis-devel] LWGEOM -- inital version ready for testing > > > There's an inital version of LWGEOM available for testing. > The indexes > it makes are about 40% smaller, and there's a very > significant reduction > of the actual geometry size. It natively supports 2D, 3D, and 4D > geometries. > > Its been tested on a Linux (Intel) machine and a Solaris (Sparc) > machine. Norman says it compiles on Windows (cygwin), so it > should also > work there. > > If you checkout a CVS copy of PostGIS, you'll see a "lwgeom/" > directory. > > There's a README file here with lots of details on how to use it. > > In summary, to test out the LWGEOM on your system: > > cd lwgeom > make > cd regress > ./run_regress > ./run_regress2 > ./run_regress3 > > > Please let me know if you're tried it out and if you had any problems. > > dave > ps. I've attached the README file - it contains more detail of how to > use the LWGEOM type. > > pps. Thanks to Ralph Mason who did lots of work doing a > lex/yacc WKT/WKB > parser and writers. > From strk at keybit.net Tue May 4 09:56:33 2004 From: strk at keybit.net (strk) Date: Tue, 4 May 2004 18:56:33 +0200 Subject: [postgis-devel] Re: Compiling postgis with gcc3.3.2 on Solaris 7 - use of -fPIC C++ compiler option In-Reply-To: <4097B84F.6050207@nlh.no>; from havard.tveite@nlh.no on Tue, May 04, 2004 at 05:35:43PM +0200 References: <4097B84F.6050207@nlh.no> Message-ID: <20040504185633.D74252@freek.keybit.net> I've added -fPIC in the Makefile, as you suggested. I don't think that will hurt. --strk; On Tue, May 04, 2004 at 05:35:43PM +0200, Havard Tveite wrote: > For your information: > > When running make, there were problems generating the shared > library (libpostgis.so.0.8). GCC failed with something like: > lots of lines of errors, and finally > "ld: fatal: relocations remain against allocatable but non-writable sections" > > I found out that if I included -fPIC for C++ in the Makefile, as > follows: > "override CXXFLAGS += -fPIC" > everything compiled fine. Without this option, the compile would > succeed when I replaced the "-shared" option with the "-G" option. > In both cases, the resulting libraries seemed to be OK, as my UMN > Mapserver application was able to get data from PostGreSQL/PostGIS. > > I am on Solaris 7, and use the Solaris /usr/ccs/bin utils. > My version of GCC is 3.3.2. > > -- > Håvard Tveite > Department of Mathematical Sciences and Technology > Agricultural University of Norway > Drøbakveien 14, POBox 5003, N-1432 Ås, NORWAY > Phone: +47 64948857 Fax: +47 64948810 http://www.nlh.no/imt From dblasby at refractions.net Tue May 4 10:34:39 2004 From: dblasby at refractions.net (David Blasby) Date: Tue, 04 May 2004 10:34:39 -0700 Subject: [postgis-devel] LWGEOM -- inital version ready for testing In-Reply-To: <8F4A22E017460A458DB7BBAB65CA6AE50265BA@openmanage> References: <8F4A22E017460A458DB7BBAB65CA6AE50265BA@openmanage> Message-ID: <4097D42F.6080905@refractions.net> Mark Cave-Ayland wrote: > Firstly a big well done for the work you've put into the LWGEOM - I had > hoped to have taken a more active role in developing the code but > unfortunately my time has been needed on other things :) Taking a look at it will help me a lot. > Anyway, haven't seen many responses from the other guys on the list so Ya - I wasnt feeling The Love... > 2. At this stage it think it would be useful to include some sort of > debugging function to > dump the LWGEOM format into a human-readable form, for example: > > > SELECT raw_lwgeom(geom) FROM geomtable; > > | size | has_srid | has_bbox | dimensionality | wkb_type | > wkt_geom | wkb_geom | > > +------+----------+----------+----------------+----------+----------+--- > -------+ > > > I can see this being useful for helping users in situations where > they are not sure > exactly how they have configured the particular geometry in the > table. What do you > think? I'll see what I can do. Its a bit tricky to have a single command return multiple columns of data, but I can easily make lwgeom_rawinfo_size()-type functions and a summary function that would give a text version of all the above info. > 3. Should we prevent users from adding bounding boxes to point columns? > (i.e. is the > single/double precision conversion fast enough to make this a waste > of disk space?) I think we should make a decision on if we're going to always have bbounding box automatically added or not. For most people, not having bounding boxes is the "best" option - the geometries are small, and simple queries arent noticably slower. For queries like: SELECT * FROM
WHERE lwgeom && ''; You will not miss the bounding boxes inside the geometries because it will be looking at the pre-generated bounding boxes in the index. Unfortunately, because of the way GiST does its searching this is actually faster: SELECT * FROM
WHERE lwgeom && AddBBox(''); Because GiST will ask for the bounding box of the search geometry many many times during the index scan (once for every level in the tree, then once for each tuple in the index leaf [about 140]). You'll probably not notice a speed difference as it usually just a few milliseconds. I tried to get GiST to pre-cache the bounding box of the search geometry, but I havent been able to do it - its a bit silly. When you start cross-joining tables - a query that does a lot of sub-index scans, adding the bounding box significantly improves performance. Crossing a 10,000 row table with itself takes about 2 second when there's bounding boxes but about 20 seconds when there's not. > 4. I think it would be useful to include BBOXes in LWGEOM by default. My > thinking here > would be that those users who would be knowlegable enough to be > concerned about the > space saving will more than likely to be knowledgable enough to > remove it where as new > users may get confused when certain queries that used to perform well > in PostGIS > perform poorly using LWGEOM. This is a good point. > 5. As far as I can see, assuming a non-index scan, the LWGEOM operators > call the box2d_* > functions directly which is defined using float4s. It looks like this > is contrary to > the OGC spec since all coordinates (and therefore I would guess > operators) are defined > as doubles? :(. > > I guess that we would need to maintain a box2d type which uses > doubles as well as > floats and use this for all the LWGEOM operators/functions (the box2d > float4 would > still be used for the indexes). Here it would be compulsory to add > RECHECK to the > operator classes since when expanding the box2d(double) to > box2d(float) extra > geometries may be returned by an overlap calculation. The RECHECK > would ensure that > these would be stripped out before the result set was returned. None of the LWGEOM operators are defined by the OGC - they're there because the GiST index needs them. When you do a " && ", you should actually be calling the GEOS "intersects(geom1,geom2)". I must admit that the only operator I've actually ever used is the "&&". The way BOX2Ds are formed, you'll always get an 'appropriate' answer. Its a bit more complex for the other operators, but you'll usually get the correct answer. If you want to do things in double-precision, you can create double-precision bounding boxes (BOX3D) from lwgeoms "box3d(lwgeom)". I understand you point, but I think it's a lot of work (mostly computation) to do things in double when the single-precision results are "good enough". If people feel strongly on this, it isnt difficult to make the change - but it will have to compute the double-precision bounding box every time since there's no way to pre-compute it. > 6. It looks like you can add bboxes and srids to a geometry but not > remove them? Ya - I need to add this ability. I'll add it once we decide if we're going to have bounding boxes by default or not. >Also if I > follow the instructions in the README to add bounding boxes to a > column: > > DROP INDEX ; > UPDATE
SET = AddBBOX(); > CREATE INDEX ON
USING GIST > ( GIST_LWGEOM_OPS); > VACUUM ANALYSE
; > > That would seem to work great. My question is now if I add more > geometries to the > table, would these new geometries not have a bounding box attached? > Would the only way > to add bounding boxes to these geometries be by running the update > query and reindexing > again? Yes - this is a problem. You'll have a mix of geometries with and without bounding boxes. > Anyway glad to see that this stuff is becoming a reality, and I hope > that this will get some discussion started on the list about your work > so far :) If I get a chance I will try and setup a test database with > some sample data and see how it performs.... You should find that it performs a wee bit slower than postgis, but it takes *significantly* less space. To find out how "big" things are: 1) vacuum analyse; --- very important! 2) SELECT relname, relpages FROM pg_class ORDER BY relpages DESC; -- gives size in disk pages (probably 8k) You'll notice that there will often be a toast table associated with a geometry table if you have "large" geometries. Thanks for the comments, looks like theres a few things to change. dave From strk at keybit.net Tue May 4 16:08:02 2004 From: strk at keybit.net (strk) Date: Wed, 5 May 2004 01:08:02 +0200 Subject: [postgis-devel] LWGEOM -- inital version ready for testing In-Reply-To: <4097D42F.6080905@refractions.net>; from dblasby@refractions.net on Tue, May 04, 2004 at 10:34:39AM -0700 References: <8F4A22E017460A458DB7BBAB65CA6AE50265BA@openmanage> <4097D42F.6080905@refractions.net> Message-ID: <20040505010802.B77518@freek.keybit.net> On Tue, May 04, 2004 at 10:34:39AM -0700, David Blasby wrote: > Mark Cave-Ayland wrote: > > > Firstly a big well done for the work you've put into the LWGEOM - I had > > hoped to have taken a more active role in developing the code but > > unfortunately my time has been needed on other things :) > > Taking a look at it will help me a lot. > > > Anyway, haven't seen many responses from the other guys on the list so > > Ya - I wasnt feeling The Love... Sorry Dave, I haven't had much time (GEOS debugging...). You did a really great job ! The first attempt at building lwgeom was successful, apart from missing prototypes in the parsers, but if I try to build it now I get a lot of errors: wktparse.lex:22: error: `lwg_parse_yylval' undeclared (first use in this functio n) wktparse.lex:22: error: (Each undeclared identifier is reported only once wktparse.lex:22: error: for each function it appears in.) wktparse.lex:22: error: `VALUE' undeclared (first use in this function) wktparse.lex:25: error: `WKB' undeclared (first use in this function) wktparse.lex:28: error: `POINT' undeclared (first use in this function) wktparse.lex:29: error: `LINESTRING' undeclared (first use in this function) wktparse.lex:30: error: `POLYGON' undeclared (first use in this function) wktparse.lex:31: error: `MULTIPOINT' undeclared (first use in this function) wktparse.lex:32: error: `MULTILINESTRING' undeclared (first use in this function) wktparse.lex:33: error: `MULTIPOLYGON' undeclared (first use in this function) wktparse.lex:34: error: `GEOMETRYCOLLECTION' undeclared (first use in this function) wktparse.lex:35: error: `SRID' undeclared (first use in this function) wktparse.lex:36: error: `EMPTY' undeclared (first use in this function) wktparse.lex:38: error: `LPAREN' undeclared (first use in this function) wktparse.lex:39: error: `RPAREN' undeclared (first use in this function) wktparse.lex:40: error: `COMMA' undeclared (first use in this function) wktparse.lex:41: error: `EQUALS' undeclared (first use in this function) wktparse.lex:42: error: `SEMICOLON' undeclared (first use in this function) Does it have to do with bison/yacc differences ? --strk; > > > 2. At this stage it think it would be useful to include some sort of > > debugging function to > > dump the LWGEOM format into a human-readable form, for example: > > > > > SELECT raw_lwgeom(geom) FROM geomtable; > > > > | size | has_srid | has_bbox | dimensionality | wkb_type | > > wkt_geom | wkb_geom | > > > > +------+----------+----------+----------------+----------+----------+--- > > -------+ > > > > > > I can see this being useful for helping users in situations where > > they are not sure > > exactly how they have configured the particular geometry in the > > table. What do you > > think? > > I'll see what I can do. Its a bit tricky to have a single command > return multiple columns of data, but I can easily make > lwgeom_rawinfo_size()-type functions and a summary function that would > give a text version of all the above info. > > > 3. Should we prevent users from adding bounding boxes to point columns? > > (i.e. is the > > single/double precision conversion fast enough to make this a waste > > of disk space?) > > I think we should make a decision on if we're going to always have > bbounding box automatically added or not. > > For most people, not having bounding boxes is the "best" option - the > geometries are small, and simple queries arent noticably slower. > > For queries like: > SELECT * FROM
WHERE lwgeom && ''; > > You will not miss the bounding boxes inside the geometries because it > will be looking at the pre-generated bounding boxes in the index. > Unfortunately, because of the way GiST does its searching this is > actually faster: > > SELECT * FROM
WHERE lwgeom && AddBBox(''); > > Because GiST will ask for the bounding box of the search geometry many > many times during the index scan (once for every level in the tree, then > once for each tuple in the index leaf [about 140]). You'll probably not > notice a speed difference as it usually just a few milliseconds. I > tried to get GiST to pre-cache the bounding box of the search geometry, > but I havent been able to do it - its a bit silly. > > > When you start cross-joining tables - a query that does a lot of > sub-index scans, adding the bounding box significantly improves > performance. Crossing a 10,000 row table with itself takes about 2 > second when there's bounding boxes but about 20 seconds when there's not. > > > 4. I think it would be useful to include BBOXes in LWGEOM by default. My > > thinking here > > would be that those users who would be knowlegable enough to be > > concerned about the > > space saving will more than likely to be knowledgable enough to > > remove it where as new > > users may get confused when certain queries that used to perform well > > in PostGIS > > perform poorly using LWGEOM. > > This is a good point. > > > 5. As far as I can see, assuming a non-index scan, the LWGEOM operators > > call the box2d_* > > functions directly which is defined using float4s. It looks like this > > is contrary to > > the OGC spec since all coordinates (and therefore I would guess > > operators) are defined > > as doubles? :(. > > > > I guess that we would need to maintain a box2d type which uses > > doubles as well as > > floats and use this for all the LWGEOM operators/functions (the box2d > > float4 would > > still be used for the indexes). Here it would be compulsory to add > > RECHECK to the > > operator classes since when expanding the box2d(double) to > > box2d(float) extra > > geometries may be returned by an overlap calculation. The RECHECK > > would ensure that > > these would be stripped out before the result set was returned. > > None of the LWGEOM operators are defined by the OGC - they're there > because the GiST index needs them. When you do a " && ", > you should actually be calling the GEOS "intersects(geom1,geom2)". > > I must admit that the only operator I've actually ever used is the "&&". > The way BOX2Ds are formed, you'll always get an 'appropriate' answer. > > Its a bit more complex for the other operators, but you'll usually get > the correct answer. > > If you want to do things in double-precision, you can create > double-precision bounding boxes (BOX3D) from lwgeoms "box3d(lwgeom)". > > I understand you point, but I think it's a lot of work (mostly > computation) to do things in double when the single-precision results > are "good enough". > > If people feel strongly on this, it isnt difficult to make the change - > but it will have to compute the double-precision bounding box every time > since there's no way to pre-compute it. > > > 6. It looks like you can add bboxes and srids to a geometry but not > > remove them? > > Ya - I need to add this ability. I'll add it once we decide if we're > going to have bounding boxes by default or not. > > >Also if I > > follow the instructions in the README to add bounding boxes to a > > column: > > > > DROP INDEX ; > > UPDATE
SET = AddBBOX(); > > CREATE INDEX ON
USING GIST > > ( GIST_LWGEOM_OPS); > > VACUUM ANALYSE
; > > > > That would seem to work great. My question is now if I add more > > geometries to the > > table, would these new geometries not have a bounding box attached? > > Would the only way > > to add bounding boxes to these geometries be by running the update > > query and reindexing > > again? > > Yes - this is a problem. You'll have a mix of geometries with and > without bounding boxes. > > > Anyway glad to see that this stuff is becoming a reality, and I hope > > that this will get some discussion started on the list about your work > > so far :) If I get a chance I will try and setup a test database with > > some sample data and see how it performs.... > > You should find that it performs a wee bit slower than postgis, but it > takes *significantly* less space. > > To find out how "big" things are: > > 1) vacuum analyse; --- very important! > 2) SELECT relname, relpages FROM pg_class ORDER BY relpages DESC; > -- gives size in disk pages (probably 8k) > You'll notice that there will often be a toast table associated with a > geometry table if you have "large" geometries. > > > Thanks for the comments, looks like theres a few things to change. > > dave > > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From Martin.Daly at cadcorp.com Thu May 6 09:15:45 2004 From: Martin.Daly at cadcorp.com (Martin Daly) Date: Thu, 6 May 2004 17:15:45 +0100 Subject: [postgis-devel] RE: Cadcorp is first GIS vendor to offer PostGIS support Message-ID: All, A quote from the press release here http://www.cadcorp.com/PostGIS.htm: "Cadcorp, developer of the pioneering Cadcorp SIS - Spatial Information System GIS software, is pleased to announce that PostGIS support is now available in Cadcorp SIS V6.1. This is the first implementation of PostGIS by a GIS vendor." More information is available here http://www.cadcorp.com/PostGISDownload.htm and here mailto:postgis at cadcorp.com Regards, Martin Daly, Technical Director, Cadcorp Computer Aided Development Corporation Ltd. www.cadcorp.com From craig.miller at spatialminds.com Thu May 6 14:10:52 2004 From: craig.miller at spatialminds.com (Craig Miller) Date: Thu, 6 May 2004 14:10:52 -0700 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs Message-ID: Hi, I am trying to compile the unsupported combination of PostgreSQL 7.5 (snapshot) and PostGIS 0.8.1 (or CVS). I know this question has come up before, but has anyone looked at the problem with the call to canonicalize_qual and determined how the call should be made to the version of this method included in pgsql 7.5? The code: /* * If the index is partial, AND the index predicate with the * explicitly given indexquals to produce a more accurate idea of the * index restriction. This may produce redundant clauses, which we * hope that cnfify and clauselist_selectivity will deal with * intelligently. * * Note that index->indpred and indexQuals are both in implicit-AND form * to start with, which we have to make explicit to hand to * canonicalize_qual, and then we get back implicit-AND form again. */ if (index->indpred != NIL) { Expr *andedQuals; andedQuals = make_ands_explicit(nconc(listCopy(index->indpred), indexQuals)); selectivityQuals = canonicalize_qual(andedQuals, true); } The error: gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declaratio ns -g -fexceptions -I. -DFRONTEND -DSYSCONFDIR='"/pgSpatial/etc/postgresql "' -DUSE_VERSION=74 -I../../src/include -I./src/include/port/win32 -DEXEC_ BACKEND "-I../../src/include/port/win32" -c -o postgis_estimate.o postgis_estimate.c postgis_estimate.c: In function `genericcostestimate2': postgis_estimate.c:751: too many arguments to function `canonicalize_qual' postgis_estimate.c:751: warning: assignment from incompatible pointer type make: *** [postgis_estimate.o] Error 1 Thanks for the help, --Craig From craig.miller at spatialminds.com Fri May 7 00:07:27 2004 From: craig.miller at spatialminds.com (Craig Miller) Date: Fri, 7 May 2004 00:07:27 -0700 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs In-Reply-To: Message-ID: Well, I didn't hear from anyone so I will see if I can find some time to poke around tomorrow and figure out the problem. --Craig -----Original Message----- From: Craig Miller [mailto:craig.miller at spatialminds.com] Sent: Thursday, May 06, 2004 2:11 PM To: postgis-devel at postgis.refractions.net Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs Hi, I am trying to compile the unsupported combination of PostgreSQL 7.5 (snapshot) and PostGIS 0.8.1 (or CVS). I know this question has come up before, but has anyone looked at the problem with the call to canonicalize_qual and determined how the call should be made to the version of this method included in pgsql 7.5? The code: /* * If the index is partial, AND the index predicate with the * explicitly given indexquals to produce a more accurate idea of the * index restriction. This may produce redundant clauses, which we * hope that cnfify and clauselist_selectivity will deal with * intelligently. * * Note that index->indpred and indexQuals are both in implicit-AND form * to start with, which we have to make explicit to hand to * canonicalize_qual, and then we get back implicit-AND form again. */ if (index->indpred != NIL) { Expr *andedQuals; andedQuals = make_ands_explicit(nconc(listCopy(index->indpred), indexQuals)); selectivityQuals = canonicalize_qual(andedQuals, true); } The error: gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declaratio ns -g -fexceptions -I. -DFRONTEND -DSYSCONFDIR='"/pgSpatial/etc/postgresql "' -DUSE_VERSION=74 -I../../src/include -I./src/include/port/win32 -DEXEC_ BACKEND "-I../../src/include/port/win32" -c -o postgis_estimate.o postgis_estimate.c postgis_estimate.c: In function `genericcostestimate2': postgis_estimate.c:751: too many arguments to function `canonicalize_qual' postgis_estimate.c:751: warning: assignment from incompatible pointer type make: *** [postgis_estimate.o] Error 1 Thanks for the help, --Craig _______________________________________________ postgis-devel mailing list postgis-devel at postgis.refractions.net http://postgis.refractions.net/mailman/listinfo/postgis-devel From m.cave-ayland at webbased.co.uk Fri May 7 01:52:45 2004 From: m.cave-ayland at webbased.co.uk (Mark Cave-Ayland) Date: Fri, 7 May 2004 09:52:45 +0100 Subject: [postgis-devel] LWGEOM -- inital version ready for testing In-Reply-To: <8F4A22E017460A458DB7BBAB65CA6AE5218A59@openmanage> Message-ID: <8F4A22E017460A458DB7BBAB65CA6AE50265C3@openmanage> Hi Dave, > Ya - I wasnt feeling The Love... Yeah, I'm quite surprised that no-one else has commented on this too.....? > I'll see what I can do. Its a bit tricky to have a single command > return multiple columns of data, but I can easily make > lwgeom_rawinfo_size()-type functions and a summary function > that would > give a text version of all the above info. Cool. > > 3. Should we prevent users from adding bounding boxes to point > > columns? (i.e. is the > > single/double precision conversion fast enough to make > this a waste > > of disk space?) > > I think we should make a decision on if we're going to always have > bbounding box automatically added or not. > > For most people, not having bounding boxes is the "best" option - the > geometries are small, and simple queries arent noticably slower. > > For queries like: > SELECT * FROM
WHERE lwgeom && ''; > > You will not miss the bounding boxes inside the geometries because it > will be looking at the pre-generated bounding boxes in the index. > Unfortunately, because of the way GiST does its searching this is > actually faster: > > SELECT * FROM
WHERE lwgeom && AddBBox(''); > > Because GiST will ask for the bounding box of the search > geometry many > many times during the index scan (once for every level in the > tree, then > once for each tuple in the index leaf [about 140]). You'll > probably not > notice a speed difference as it usually just a few milliseconds. I > tried to get GiST to pre-cache the bounding box of the search > geometry, > but I havent been able to do it - its a bit silly. > > > When you start cross-joining tables - a query that does a lot of > sub-index scans, adding the bounding box significantly improves > performance. Crossing a 10,000 row table with itself takes about 2 > second when there's bounding boxes but about 20 seconds when > there's not. Ah I see, although this wasn't what I was thinking when I wrote the email! I was thinking about that since a point has its own bounding box then would it be a waste of space to specify the same information contained with the point from the bounding box. (cut) > > 5. As far as I can see, assuming a non-index scan, the LWGEOM > > operators call the box2d_* > > functions directly which is defined using float4s. It looks like > > this is contrary to > > the OGC spec since all coordinates (and therefore I would guess > > operators) are defined > > as doubles? :(. > > > > I guess that we would need to maintain a box2d type which uses > > doubles as well as > > floats and use this for all the LWGEOM operators/functions (the > > box2d float4 would > > still be used for the indexes). Here it would be > compulsory to add > > RECHECK to the > > operator classes since when expanding the box2d(double) to > > box2d(float) extra > > geometries may be returned by an overlap calculation. The RECHECK > > would ensure that > > these would be stripped out before the result set was returned. > > None of the LWGEOM operators are defined by the OGC - they're there > because the GiST index needs them. When you do a " && > ", > you should actually be calling the GEOS "intersects(geom1,geom2)". > > I must admit that the only operator I've actually ever used > is the "&&". > The way BOX2Ds are formed, you'll always get an > 'appropriate' answer. > > Its a bit more complex for the other operators, but you'll > usually get > the correct answer. > > If you want to do things in double-precision, you can create > double-precision bounding boxes (BOX3D) from lwgeoms "box3d(lwgeom)". > > I understand you point, but I think it's a lot of work (mostly > computation) to do things in double when the single-precision results > are "good enough". > > If people feel strongly on this, it isnt difficult to make > the change - > but it will have to compute the double-precision bounding box > every time > since there's no way to pre-compute it. My only concern would that LWGEOM would fail an OGC regression test that used data up to the full precision of a double. I also know that some of our lat/long datasets can have precision because they were geocoded against hi-resolution raster imagery. Although with LWGEOM, I thought that internally everything would still be a double except for the bounding boxes in the GiST index? Then again, thinking about it now, I suppose we would need the 'true' double precision bounding box stored in the actual geometry for the RECHECK operator. > You should find that it performs a wee bit slower than > postgis, but it > takes *significantly* less space. Interesting. So while it may be slightly slower to begin with, it should still scale better for the reason that there is less data to pull off disk, no? Cheers, Mark. --- Mark Cave-Ayland Webbased Ltd. Tamar Science Park Derriford Plymouth PL6 8BX England Tel: +44 (0)1752 764445 Fax: +44 (0)1752 764446 This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. From m.cave-ayland at webbased.co.uk Fri May 7 02:43:32 2004 From: m.cave-ayland at webbased.co.uk (Mark Cave-Ayland) Date: Fri, 7 May 2004 10:43:32 +0100 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs In-Reply-To: <8F4A22E017460A458DB7BBAB65CA6AE5218B60@openmanage> Message-ID: <8F4A22E017460A458DB7BBAB65CA6AE50265C4@openmanage> Hi Craig, A few of us have been using the CVS version of PostGIS with PostgreSQL 7.5 in order to improve the statistics code to make use of some new functionality - I think strk put the relevant changes into the CVS code. I haven't used a 7.5/PostGIS combo for a couple of months but can't think of any recent changes that would break anything. Hope this helps, Mark. --- Mark Cave-Ayland Webbased Ltd. Tamar Science Park Derriford Plymouth PL6 8BX England Tel: +44 (0)1752 764445 Fax: +44 (0)1752 764446 This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. > -----Original Message----- > From: postgis-devel-bounces at postgis.refractions.net > [mailto:postgis-devel-bounces at postgis.refractions.net] On > Behalf Of Craig Miller > Sent: 07 May 2004 08:07 > To: PostGIS Development Discussion > Subject: RE: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs > > > Well, I didn't hear from anyone so I will see if I can find > some time to poke around tomorrow and figure out the problem. > > --Craig > > > -----Original Message----- > From: Craig Miller [mailto:craig.miller at spatialminds.com] > Sent: Thursday, May 06, 2004 2:11 PM > To: postgis-devel at postgis.refractions.net > Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs > > > Hi, > > I am trying to compile the unsupported combination of PostgreSQL 7.5 > (snapshot) and PostGIS 0.8.1 (or CVS). I know this question > has come up before, but has anyone looked at the problem with > the call to canonicalize_qual and determined how the call > should be made to the version of this method included in pgsql 7.5? > > The code: > /* > * If the index is partial, AND the index predicate with the > * explicitly given indexquals to produce a more > accurate idea of the > * index restriction. This may produce redundant > clauses, which we > * hope that cnfify and clauselist_selectivity will deal with > * intelligently. > * > * Note that index->indpred and indexQuals are both > in implicit-AND form > * to start with, which we have to make explicit to hand to > * canonicalize_qual, and then we get back > implicit-AND form again. > */ > if (index->indpred != NIL) > { > Expr *andedQuals; > > andedQuals = > make_ands_explicit(nconc(listCopy(index->indpred), > > indexQuals)); > selectivityQuals = > canonicalize_qual(andedQuals, true); > } > > > The error: > > gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes > -Wmissing-declaratio > ns -g -fexceptions -I. -DFRONTEND > -DSYSCONFDIR='"/pgSpatial/etc/postgresql > "' -DUSE_VERSION=74 -I../../src/include > -I./src/include/port/win32 -DEXEC_ BACKEND > "-I../../src/include/port/win32" -c -o postgis_estimate.o > postgis_estimate.c > > postgis_estimate.c: In function `genericcostestimate2': > postgis_estimate.c:751: too many arguments to function > `canonicalize_qual' > postgis_estimate.c:751: warning: assignment from incompatible > pointer type > make: *** [postgis_estimate.o] Error 1 > > Thanks for the help, > --Craig > > > _______________________________________________ > postgis-devel mailing list postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel > > > > > _______________________________________________ > postgis-devel mailing list postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel > From strk at keybit.net Fri May 7 03:50:59 2004 From: strk at keybit.net (strk) Date: Fri, 7 May 2004 12:50:59 +0200 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs In-Reply-To: ; from craig.miller@spatialminds.com on Thu, May 06, 2004 at 02:10:52PM -0700 References: Message-ID: <20040507125059.A2093@freek.keybit.net> On Thu, May 06, 2004 at 02:10:52PM -0700, Craig Miller wrote: > Hi, > > I am trying to compile the unsupported combination of PostgreSQL 7.5 > (snapshot) and PostGIS 0.8.1 (or CVS). I know this question has come up > before, but has anyone looked at the problem with the call to > canonicalize_qual and determined how the call should be made to the version > of this method included in pgsql 7.5? > > The code: > /* > * If the index is partial, AND the index predicate with the > * explicitly given indexquals to produce a more accurate idea of > the > * index restriction. This may produce redundant clauses, which we > * hope that cnfify and clauselist_selectivity will deal with > * intelligently. > * > * Note that index->indpred and indexQuals are both in implicit-AND > form > * to start with, which we have to make explicit to hand to > * canonicalize_qual, and then we get back implicit-AND form again. > */ > if (index->indpred != NIL) > { > Expr *andedQuals; > > andedQuals = > make_ands_explicit(nconc(listCopy(index->indpred), > > indexQuals)); > selectivityQuals = canonicalize_qual(andedQuals, true); > } > > > The error: > > gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declaratio > ns -g -fexceptions -I. -DFRONTEND -DSYSCONFDIR='"/pgSpatial/etc/postgresql > "' -DUSE_VERSION=74 -I../../src/include -I./src/include/port/win32 -DEXEC_ > BACKEND "-I../../src/include/port/win32" -c -o postgis_estimate.o > postgis_estimate.c > > postgis_estimate.c: In function `genericcostestimate2': > postgis_estimate.c:751: too many arguments to function `canonicalize_qual' > postgis_estimate.c:751: warning: assignment from incompatible pointer type > make: *** [postgis_estimate.o] Error 1 You are compiling against pg74, not 75 (-DUSE_VERSION=74). To detect pgsql version, postgis Makefile goes: ifneq ($(findstring 7.4,$(VERSION)),) USE_VERSION=74 else USE_VERSION=75 endif What does your postgres snapshot's global Makefile use as $(VERSION) ? --strk; > > Thanks for the help, > --Craig > > > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From dblasby at refractions.net Fri May 7 09:40:07 2004 From: dblasby at refractions.net (David Blasby) Date: Fri, 07 May 2004 09:40:07 -0700 Subject: [postgis-devel] LWGEOM -- inital version ready for testing In-Reply-To: <8F4A22E017460A458DB7BBAB65CA6AE50265C3@openmanage> References: <8F4A22E017460A458DB7BBAB65CA6AE50265C3@openmanage> Message-ID: <409BBBE7.80208@refractions.net> Mark Cave-Ayland wrote: > Ah I see, although this wasn't what I was thinking when I wrote the > email! I was thinking about that since a point has its own bounding box > then would it be a waste of space to specify the same information > contained with the point from the bounding box. True, but there is still significant overhead of looking at the geometry, checking to see if its a point, then constructing the BOX3D, then converting it to a BOX2DFLOAT4. It not much overhead, but it adds up for the nested queries. > My only concern would that LWGEOM would fail an OGC regression test that > used data up to the full precision of a double. I also know that some of > our lat/long datasets can have precision because they were geocoded > against hi-resolution raster imagery. Although with LWGEOM, I thought > that internally everything would still be a double except for the > bounding boxes in the GiST index? Then again, thinking about it now, I > suppose we would need the 'true' double precision bounding box stored in > the actual geometry for the RECHECK operator. None of the operators are in the OGC spec. The one's I'm talking about are "&&", and the almost-never used ones like '<&'. Personally, I'd like to see all but && and maybe contains/contained removed. If you were to do a intersects(g1,g2), you'll be using the actual double-precision coordinates. The idea of the operators is to do two-stage queries: SELECT ... FROM
WHERE g && AND ; As long as the first stage (&&) give you correct results, the 2nd stage will always work. The way I construct the box2d, the && operator will never give you a false negative. >>You should find that it performs a wee bit slower than >>postgis, but it >>takes *significantly* less space. > > > Interesting. So while it may be slightly slower to begin with, it should > still scale better for the reason that there is less data to pull off > disk, no? Well - this depends on how well your system disk cache is working. Its really hard to test true speed because it takes a LONG time to be reasonably sure the cache is empty (know any way to flush the cache under linux?). I'm just in process of creating a table with 3 geometry columns in it - a 2 point LINESTRING and two points. There's going to be 17,000,000 entries in it. Under PostGIS, the table would take 7.4 Gb. Under LWGEOM, it only takes a little over 1Gb. You can imagine how much faster a sequential scan would go on the LWGEOM. For index searches, it much more likely that the disk cache would actually help. If the LWGEOM stuff was a little more tested, I'd use it - it takes 4 days of processing to make the table, so I dont want to screw it up. dave From craig.miller at spatialminds.com Fri May 7 14:00:47 2004 From: craig.miller at spatialminds.com (Craig Miller) Date: Fri, 7 May 2004 14:00:47 -0700 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs In-Reply-To: <20040507125059.A2093@freek.keybit.net> Message-ID: I am building against 7.5devel. Postgis 0.8.1's Makefile defaults to 7.4 rather than 7.5. I will grab a postgis snapshot from CVS and see if I can get that to build. Glad to hear the code is in CVS already. Thanks! --Craig -----Original Message----- From: strk [mailto:strk at keybit.net] Sent: Friday, May 07, 2004 3:51 AM To: Craig Miller Cc: postgis-devel at postgis.refractions.net Subject: Re: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs On Thu, May 06, 2004 at 02:10:52PM -0700, Craig Miller wrote: > Hi, > > I am trying to compile the unsupported combination of PostgreSQL 7.5 > (snapshot) and PostGIS 0.8.1 (or CVS). I know this question has come up > before, but has anyone looked at the problem with the call to > canonicalize_qual and determined how the call should be made to the version > of this method included in pgsql 7.5? > > The code: > /* > * If the index is partial, AND the index predicate with the > * explicitly given indexquals to produce a more accurate idea of > the > * index restriction. This may produce redundant clauses, which we > * hope that cnfify and clauselist_selectivity will deal with > * intelligently. > * > * Note that index->indpred and indexQuals are both in implicit-AND > form > * to start with, which we have to make explicit to hand to > * canonicalize_qual, and then we get back implicit-AND form again. > */ > if (index->indpred != NIL) > { > Expr *andedQuals; > > andedQuals = > make_ands_explicit(nconc(listCopy(index->indpred), > > indexQuals)); > selectivityQuals = canonicalize_qual(andedQuals, true); > } > > > The error: > > gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declaratio > ns -g -fexceptions -I. -DFRONTEND -DSYSCONFDIR='"/pgSpatial/etc/postgresql > "' -DUSE_VERSION=74 -I../../src/include -I./src/include/port/win32 -DEXEC_ > BACKEND "-I../../src/include/port/win32" -c -o postgis_estimate.o > postgis_estimate.c > > postgis_estimate.c: In function `genericcostestimate2': > postgis_estimate.c:751: too many arguments to function `canonicalize_qual' > postgis_estimate.c:751: warning: assignment from incompatible pointer type > make: *** [postgis_estimate.o] Error 1 You are compiling against pg74, not 75 (-DUSE_VERSION=74). To detect pgsql version, postgis Makefile goes: ifneq ($(findstring 7.4,$(VERSION)),) USE_VERSION=74 else USE_VERSION=75 endif What does your postgres snapshot's global Makefile use as $(VERSION) ? --strk; > > Thanks for the help, > --Craig > > > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From craig.miller at spatialminds.com Fri May 7 15:59:27 2004 From: craig.miller at spatialminds.com (Craig Miller) Date: Fri, 7 May 2004 15:59:27 -0700 Subject: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs In-Reply-To: <20040507125059.A2093@freek.keybit.net> Message-ID: I am getting closer. I am only having linker errors now. In case anyone else wants to know what I did. After getting the latest from CVS I first applied the changes to the Makefile and postgis.h that others have already described. I then tried to compile and received an error about using index and rindex in postgis_inout.c. It turns out that these are non-ANSI versions of strchr and strrchr respectivley. msvcrt.dll exports only the ANSI names, so I changed the references to the non-ANSI names to strchr and strrchr respectively. E.g. they now read: first_paren= strchr (str,'('); last_paren = strrchr(str,')'); So now I am left with the following linker errors. To be frank, I have not used Make, autoconf, automake, etc in many years (too much Visual C++ and Java lateley) so I am having trouble figuring out how to correct this "correctly". If anyone has info on the correct way to fix this, that would be appreciated. If not, I have my GNU Autoconf, automak, and libtool book cracked open and should be able to get it figured out before too long. Here is the link error I still have left. dlltool --export-all --output-def postgis.def postgis_debug.o postgis_ops.o postgis_fn.o postgis_inout.o postgis_proj.o postgis_chip.o postgis_transform.o postgis_gist_72.o postgis_estimate.o postgis_geos.o postgis_algo.o dllwrap -o libpostgis.dll --dllname libpostgis.dll --def postgis.def postgis_debug.o postgis_ops.o postgis_fn.o postgis_inout.o postgis_proj.o postgis_chip.o postgis_transform.o postgis_gist_72.o postgis_estimate.o postgis_geos.o postgis_algo.o -L../../src/port -L../../src/port -L../../src/backend -lpost gres Info: resolving _default_statistics_target by linking to __imp__default_statistics_target (auto-import) fu000001.o(.idata$3+0xc): undefined reference to `libpostgres_a_iname' nmth000000.o(.idata$4+0x0): undefined reference to `_nm__default_statistics_target' c:\mingw\bin\dllwrap.exe: c:\mingw\bin\gcc exited with status 1 make: *** [libpostgis.a] Error 1 --Craig -----Original Message----- From: strk [mailto:strk at keybit.net] Sent: Friday, May 07, 2004 3:51 AM To: Craig Miller Cc: postgis-devel at postgis.refractions.net Subject: Re: [postgis-devel] PostgreSQL 7.5 and PostGIS 0.8.1 or cvs On Thu, May 06, 2004 at 02:10:52PM -0700, Craig Miller wrote: > Hi, > > I am trying to compile the unsupported combination of PostgreSQL 7.5 > (snapshot) and PostGIS 0.8.1 (or CVS). I know this question has come up > before, but has anyone looked at the problem with the call to > canonicalize_qual and determined how the call should be made to the version > of this method included in pgsql 7.5? > > The code: > /* > * If the index is partial, AND the index predicate with the > * explicitly given indexquals to produce a more accurate idea of > the > * index restriction. This may produce redundant clauses, which we > * hope that cnfify and clauselist_selectivity will deal with > * intelligently. > * > * Note that index->indpred and indexQuals are both in implicit-AND > form > * to start with, which we have to make explicit to hand to > * canonicalize_qual, and then we get back implicit-AND form again. > */ > if (index->indpred != NIL) > { > Expr *andedQuals; > > andedQuals = > make_ands_explicit(nconc(listCopy(index->indpred), > > indexQuals)); > selectivityQuals = canonicalize_qual(andedQuals, true); > } > > > The error: > > gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declaratio > ns -g -fexceptions -I. -DFRONTEND -DSYSCONFDIR='"/pgSpatial/etc/postgresql > "' -DUSE_VERSION=74 -I../../src/include -I./src/include/port/win32 -DEXEC_ > BACKEND "-I../../src/include/port/win32" -c -o postgis_estimate.o > postgis_estimate.c > > postgis_estimate.c: In function `genericcostestimate2': > postgis_estimate.c:751: too many arguments to function `canonicalize_qual' > postgis_estimate.c:751: warning: assignment from incompatible pointer type > make: *** [postgis_estimate.o] Error 1 You are compiling against pg74, not 75 (-DUSE_VERSION=74). To detect pgsql version, postgis Makefile goes: ifneq ($(findstring 7.4,$(VERSION)),) USE_VERSION=74 else USE_VERSION=75 endif What does your postgres snapshot's global Makefile use as $(VERSION) ? --strk; > > Thanks for the help, > --Craig > > > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From strk at keybit.net Wed May 12 07:10:25 2004 From: strk at keybit.net (strk) Date: Wed, 12 May 2004 16:10:25 +0200 Subject: [postgis-devel] Re: dbfopen numeric fields handling broken ? In-Reply-To: <40A22D19.1060903@pobox.com>; from warmerdam@pobox.com on Wed, May 12, 2004 at 09:56:41AM -0400 References: <20040512121347.A91720@freek.keybit.net> <40A2183A.2020606@pobox.com> <20040512144043.A93029@freek.keybit.net> <40A22D19.1060903@pobox.com> Message-ID: <20040512161025.A96711@freek.keybit.net> On Wed, May 12, 2004 at 09:56:41AM -0400, Frank Warmerdam wrote: > strk wrote: > > On Wed, May 12, 2004 at 08:27:38AM -0400, Frank Warmerdam wrote: > > > >>strk wrote: > >> > >>>Hello again Frank. > >>>I got a .dbf containing a Numeric field with lenght=12 and > >>>decimal places=0. > >>>DBFGetFieldInfo() tells me it's a FTInteger (because of 0 decimals). > >>>DBFReadIntegerAttribute() casts the value to an integer, but this > >>>conversion seems to be inadeguate, since I get always -2147483648. > >>>Values goes from 10010001000 to 51580035000 (viewing with dbview), > >>>so it does not seem to be a corrupted dataset. > >>> > >>>dbfopen.c is Revision 1.52 2003/07/08 15:20:03 warmerda. > >>> > >>>Any hint on how to fix this ? > >> > >>Strk, > >> > >>Can you provide the .dbf file? I have heard other reports of problems with > >>large integer values and I would like to try and resolve the issue. > > > > > > I've checked and fixed the bug already. > > What is needed is have DBFReadIntegerAttribute return an int with 64bits, > > or directly return a double if an int64 is not enough to contain the > > maximum lenght for a dbf field of type N with no digits. > > For sure returning 'int' (signed) does not allow numbers higher > > then 2147483647 or -2147483648, whih is 9 digits guaranteed. No more. > > Strk, > > Can you provide a sample dbf file demonstrating the problem? Sorry, but I can't. I dunno how. I tried exporting a postgis table, but it seems that the same problem with Reading is there when Writing, probably DBFWriteIntegerAttribute... If you have tools to produce a dbf should be really straightforward to reproduce the bug: create a field of type N with 12 digits and 0 decimal places. Put the value 620570012000 in the field. Read it. Write it in a new dbf. Check the new dbf. I've included postgis-devel in the recipient list since this bug makes both postgis loader and dumper bogus as well.. --strk; > > Best regards, > -- > ---------------------------------------+-------------------------------------- > I set the clouds in motion - turn up | Frank Warmerdam, warmerdam at pobox.com > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush | Geospatial Programmer for Rent From ri at ita.br Wed May 12 08:07:56 2004 From: ri at ita.br (Rogerio Ishibashi) Date: Wed, 12 May 2004 12:07:56 -0300 Subject: [postgis-devel] Geotools with Postgis References: <20040512121347.A91720@freek.keybit.net><40A2183A.2020606@pobox.com><20040512144043.A93029@freek.keybit.net><40A22D19.1060903@pobox.com> <20040512161025.A96711@freek.keybit.net> Message-ID: <003001c43832$e7c25750$660918a1@fcmf.ita.br> Have someone a example of geotools with postigis?? I need a example that show data (from postigis) in a map (in a java Frame) []'s Rogerio Ishibashi From cspoerri at cavtel.net Wed May 12 20:17:42 2004 From: cspoerri at cavtel.net (Christoph Spoerri) Date: Wed, 12 May 2004 23:17:42 -0400 Subject: [postgis-devel] use of palloc() crashes server Message-ID: <200405122317.42142.cspoerri@cavtel.net> Hi there, I'm working on a couple new function for PostGIS (dyn. segmentation/linear referencing system). I ran now into the problem where postgresql crashes when I'm calling line_make(). I traced it down to the palloc() call within the line_make() function, but I have no idea what could cause the error. The line I'm creating has around 370 points, and the lines where the code crashes is: result= (LINE3D *) palloc (*size); (in file: postgis_inout.c) And just to make the problem more confusing: I run the following query (after logging in): select lrs_getshape(wkb_geometry, 1, 80) from test; => code crashes yet, if I run the following sequence: select lrs_getshape(wkb_geometry, 1, 20) from test; select lrs_getshape(wkb_geometry, 1, 80) from test; => code doesn't crash (????) It crashes again if I run the following sequence: select lrs_getshape(wkb_geometry, 1, 20) from test; select lrs_getshape(wkb_geometry, 1, 80) from test; select lrs_getshape(wkb_geometry, 1, 90) from test; Note: lrs_getshape() accumulates the necessary points to create a new 3D line. Any idea what could cause this? And thanks for your help, Christoph From strk at keybit.net Wed May 12 23:32:30 2004 From: strk at keybit.net (strk) Date: Thu, 13 May 2004 08:32:30 +0200 Subject: [postgis-devel] use of palloc() crashes server In-Reply-To: <200405122317.42142.cspoerri@cavtel.net>; from cspoerri@cavtel.net on Wed, May 12, 2004 at 11:17:42PM -0400 References: <200405122317.42142.cspoerri@cavtel.net> Message-ID: <20040513083230.A1693@freek.keybit.net> On Wed, May 12, 2004 at 11:17:42PM -0400, Christoph Spoerri wrote: > Hi there, > > I'm working on a couple new function for PostGIS (dyn. segmentation/linear > referencing system). I ran now into the problem where postgresql crashes when > I'm calling line_make(). I traced it down to the palloc() call within the > line_make() function, but I have no idea what could cause the error. > > The line I'm creating has around 370 points, and the lines where the code > crashes is: > result= (LINE3D *) palloc (*size); (in file: postgis_inout.c) > > And just to make the problem more confusing: > > I run the following query (after logging in): > select lrs_getshape(wkb_geometry, 1, 80) from test; > => code crashes > > yet, if I run the following sequence: > select lrs_getshape(wkb_geometry, 1, 20) from test; > select lrs_getshape(wkb_geometry, 1, 80) from test; > => code doesn't crash (????) > > It crashes again if I run the following sequence: > select lrs_getshape(wkb_geometry, 1, 20) from test; > select lrs_getshape(wkb_geometry, 1, 80) from test; > select lrs_getshape(wkb_geometry, 1, 90) from test; > > Note: lrs_getshape() accumulates the necessary points to create a new 3D line. > > Any idea what could cause this? And thanks for your help, What you describe is a tipical behaviour of pointers mess. There's probably *always* an out-of-bounday condition, but sometimes the faulty memory accessed is still allocated for the process, sometimes not. Try CVS postgis if possible. --strk; > Christoph > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From strk at keybit.net Thu May 13 01:20:08 2004 From: strk at keybit.net (strk) Date: Thu, 13 May 2004 10:20:08 +0200 Subject: [postgis-devel] Re: dbfopen numeric fields handling broken ? In-Reply-To: <20040512161025.A96711@freek.keybit.net>; from strk@keybit.net on Wed, May 12, 2004 at 04:10:25PM +0200 References: <20040512121347.A91720@freek.keybit.net> <40A2183A.2020606@pobox.com> <20040512144043.A93029@freek.keybit.net> <40A22D19.1060903@pobox.com> <20040512161025.A96711@freek.keybit.net> Message-ID: <20040513102008.B5716@freek.keybit.net> I've made shp2pgsql always read dbf attributes as strings, letting postgres do appropriate transformation. This seems a good workaround for the 64bit integers missing support in shapelib. Dumping, on the other hand, is not so easy, since shapelib does require a number for a field of numeric type, and then does string conversion internally, discarding higher bits. A solution would be to call DBFWriteAttributeDirectly, but then the dumper should care about formatting NULL values in the way the dbf expect that... --strk; On Wed, May 12, 2004 at 04:10:25PM +0200, strk wrote: > On Wed, May 12, 2004 at 09:56:41AM -0400, Frank Warmerdam wrote: > > strk wrote: > > > On Wed, May 12, 2004 at 08:27:38AM -0400, Frank Warmerdam wrote: > > > > > >>strk wrote: > > >> > > >>>Hello again Frank. > > >>>I got a .dbf containing a Numeric field with lenght=12 and > > >>>decimal places=0. > > >>>DBFGetFieldInfo() tells me it's a FTInteger (because of 0 decimals). > > >>>DBFReadIntegerAttribute() casts the value to an integer, but this > > >>>conversion seems to be inadeguate, since I get always -2147483648. > > >>>Values goes from 10010001000 to 51580035000 (viewing with dbview), > > >>>so it does not seem to be a corrupted dataset. > > >>> > > >>>dbfopen.c is Revision 1.52 2003/07/08 15:20:03 warmerda. > > >>> > > >>>Any hint on how to fix this ? > > >> > > >>Strk, > > >> > > >>Can you provide the .dbf file? I have heard other reports of problems with > > >>large integer values and I would like to try and resolve the issue. > > > > > > > > > I've checked and fixed the bug already. > > > What is needed is have DBFReadIntegerAttribute return an int with 64bits, > > > or directly return a double if an int64 is not enough to contain the > > > maximum lenght for a dbf field of type N with no digits. > > > For sure returning 'int' (signed) does not allow numbers higher > > > then 2147483647 or -2147483648, whih is 9 digits guaranteed. No more. > > > > Strk, > > > > Can you provide a sample dbf file demonstrating the problem? > > Sorry, but I can't. > I dunno how. > I tried exporting a postgis table, but it seems that the same > problem with Reading is there when Writing, probably > DBFWriteIntegerAttribute... > If you have tools to produce a dbf should be really straightforward > to reproduce the bug: create a field of type N with 12 digits and 0 > decimal places. Put the value 620570012000 in the field. > Read it. > Write it in a new dbf. > Check the new dbf. > > I've included postgis-devel in the recipient list since this bug > makes both postgis loader and dumper bogus as well.. > > --strk; > > > > > Best regards, > > -- > > ---------------------------------------+-------------------------------------- > > I set the clouds in motion - turn up | Frank Warmerdam, warmerdam at pobox.com > > light and sound - activate the windows | http://pobox.com/~warmerdam > > and watch the world go round - Rush | Geospatial Programmer for Rent > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From strk at keybit.net Thu May 13 05:23:35 2004 From: strk at keybit.net (strk) Date: Thu, 13 May 2004 14:23:35 +0200 Subject: [postgis-devel] Re: dbfopen numeric fields handling broken ? In-Reply-To: <20040513102008.B5716@freek.keybit.net>; from strk@keybit.net on Thu, May 13, 2004 at 10:20:08AM +0200 References: <20040512121347.A91720@freek.keybit.net> <40A2183A.2020606@pobox.com> <20040512144043.A93029@freek.keybit.net> <40A22D19.1060903@pobox.com> <20040512161025.A96711@freek.keybit.net> <20040513102008.B5716@freek.keybit.net> Message-ID: <20040513142335.A8019@freek.keybit.net> On Thu, May 13, 2004 at 10:20:08AM +0200, strk wrote: > I've made shp2pgsql always read dbf attributes as strings, letting > postgres do appropriate transformation. This seems a good workaround > for the 64bit integers missing support in shapelib. > > Dumping, on the other hand, is not so easy, since shapelib does > require a number for a field of numeric type, and then does string > conversion internally, discarding higher bits. > A solution would be to call DBFWriteAttributeDirectly, but then > the dumper should care about formatting NULL values in the way > the dbf expect that... I've checked, and actually current behaviour is not to handle NULL values specially. They are just transformed to 0 if number and empty string if strings. So I've changed pgsql2shp.c to use DBFWriteAttributeDirectly(), which does not attempt at any conversion. As before NULL numbers are transformed to 0. --strk; > > --strk; > > On Wed, May 12, 2004 at 04:10:25PM +0200, strk wrote: > > On Wed, May 12, 2004 at 09:56:41AM -0400, Frank Warmerdam wrote: > > > strk wrote: > > > > On Wed, May 12, 2004 at 08:27:38AM -0400, Frank Warmerdam wrote: > > > > > > > >>strk wrote: > > > >> > > > >>>Hello again Frank. > > > >>>I got a .dbf containing a Numeric field with lenght=12 and > > > >>>decimal places=0. > > > >>>DBFGetFieldInfo() tells me it's a FTInteger (because of 0 decimals). > > > >>>DBFReadIntegerAttribute() casts the value to an integer, but this > > > >>>conversion seems to be inadeguate, since I get always -2147483648. > > > >>>Values goes from 10010001000 to 51580035000 (viewing with dbview), > > > >>>so it does not seem to be a corrupted dataset. > > > >>> > > > >>>dbfopen.c is Revision 1.52 2003/07/08 15:20:03 warmerda. > > > >>> > > > >>>Any hint on how to fix this ? > > > >> > > > >>Strk, > > > >> > > > >>Can you provide the .dbf file? I have heard other reports of problems with > > > >>large integer values and I would like to try and resolve the issue. > > > > > > > > > > > > I've checked and fixed the bug already. > > > > What is needed is have DBFReadIntegerAttribute return an int with 64bits, > > > > or directly return a double if an int64 is not enough to contain the > > > > maximum lenght for a dbf field of type N with no digits. > > > > For sure returning 'int' (signed) does not allow numbers higher > > > > then 2147483647 or -2147483648, whih is 9 digits guaranteed. No more. > > > > > > Strk, > > > > > > Can you provide a sample dbf file demonstrating the problem? > > > > Sorry, but I can't. > > I dunno how. > > I tried exporting a postgis table, but it seems that the same > > problem with Reading is there when Writing, probably > > DBFWriteIntegerAttribute... > > If you have tools to produce a dbf should be really straightforward > > to reproduce the bug: create a field of type N with 12 digits and 0 > > decimal places. Put the value 620570012000 in the field. > > Read it. > > Write it in a new dbf. > > Check the new dbf. > > > > I've included postgis-devel in the recipient list since this bug > > makes both postgis loader and dumper bogus as well.. > > > > --strk; > > > > > > > > Best regards, > > > -- > > > ---------------------------------------+-------------------------------------- > > > I set the clouds in motion - turn up | Frank Warmerdam, warmerdam at pobox.com > > > light and sound - activate the windows | http://pobox.com/~warmerdam > > > and watch the world go round - Rush | Geospatial Programmer for Rent > > _______________________________________________ > > postgis-devel mailing list > > postgis-devel at postgis.refractions.net > > http://postgis.refractions.net/mailman/listinfo/postgis-devel > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From cspoerri at cavtel.net Thu May 13 20:12:14 2004 From: cspoerri at cavtel.net (Christoph Spoerri) Date: Thu, 13 May 2004 23:12:14 -0400 Subject: [postgis-devel] use of palloc() crashes server In-Reply-To: <20040513083230.A1693@freek.keybit.net> References: <200405122317.42142.cspoerri@cavtel.net> <20040513083230.A1693@freek.keybit.net> Message-ID: <200405132312.14064.cspoerri@cavtel.net> I have updated my code from the CVS (postgis), but encounter still the same problem. Is this a document problem or has it to do with my code? Is there a work around about the problem? Thanks, Christoph On Thursday 13 May 2004 02:32, strk wrote: > On Wed, May 12, 2004 at 11:17:42PM -0400, Christoph Spoerri wrote: > > Hi there, > > > > I'm working on a couple new function for PostGIS (dyn. > > segmentation/linear referencing system). I ran now into the problem where > > postgresql crashes when I'm calling line_make(). I traced it down to the > > palloc() call within the line_make() function, but I have no idea what > > could cause the error. > > > > The line I'm creating has around 370 points, and the lines where the code > > crashes is: > > result= (LINE3D *) palloc (*size); (in file: postgis_inout.c) > > > > And just to make the problem more confusing: > > > > I run the following query (after logging in): > > select lrs_getshape(wkb_geometry, 1, 80) from test; > > => code crashes > > > > yet, if I run the following sequence: > > select lrs_getshape(wkb_geometry, 1, 20) from test; > > select lrs_getshape(wkb_geometry, 1, 80) from test; > > => code doesn't crash (????) > > > > It crashes again if I run the following sequence: > > select lrs_getshape(wkb_geometry, 1, 20) from test; > > select lrs_getshape(wkb_geometry, 1, 80) from test; > > select lrs_getshape(wkb_geometry, 1, 90) from test; > > > > Note: lrs_getshape() accumulates the necessary points to create a new 3D > > line. > > > > Any idea what could cause this? And thanks for your help, > > What you describe is a tipical behaviour of pointers mess. > There's probably *always* an out-of-bounday condition, but > sometimes the faulty memory accessed is still allocated for > the process, sometimes not. > > Try CVS postgis if possible. > > --strk; > > > Christoph > > _______________________________________________ > > postgis-devel mailing list > > postgis-devel at postgis.refractions.net > > http://postgis.refractions.net/mailman/listinfo/postgis-devel > > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel From dblasby at refractions.net Thu May 13 20:31:10 2004 From: dblasby at refractions.net (Dave Blasby) Date: Thu, 13 May 2004 20:31:10 -0700 Subject: [postgis-devel] use of palloc() crashes server In-Reply-To: <200405132312.14064.cspoerri@cavtel.net> References: <200405122317.42142.cspoerri@cavtel.net> <20040513083230.A1693@freek.keybit.net> <200405132312.14064.cspoerri@cavtel.net> Message-ID: <1084505470.40a43d7eb0a9c@hydra> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From carl.anderson at vadose.org Thu May 20 20:07:57 2004 From: carl.anderson at vadose.org (Carl Anderson) Date: Fri, 21 May 2004 03:07:57 +0000 Subject: [postgis-devel] schema aware functions Message-ID: <1085108877l.21775l.0l@vadose.dvg> for consideration for schema aware Postgresql versions update geometry is a replacement build_histogram2d is a new fcn (new prototype) find_extent is a new fcn (new prototype) It works for me (TM) the existing functions presume that all geometry tables are accesible through the search_path. These fcns are based on f_table_schema. f_table_name. -------------- next part -------------- A non-text attachment was scrubbed... Name: build_histogram2d.sql Type: text/x-sql Size: 581 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: find_extent.sql Type: text/x-sql Size: 514 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: update_geometry_stats.sql Type: text/x-sql Size: 1682 bytes Desc: not available URL: From ithaqua at attglobal.net Fri May 21 05:37:42 2004 From: ithaqua at attglobal.net (ithaqua) Date: Fri, 21 May 2004 07:37:42 -0500 Subject: [postgis-devel] Geotools with Postgis In-Reply-To: <003001c43832$e7c25750$660918a1@fcmf.ita.br> References: <20040512121347.A91720@freek.keybit.net> <40A2183A.2020606@pobox.com><20040512144043.A93029@freek.keybit.net> <40A22D19.1060903@pobox.com> <20040512161025.A96711@freek.keybit.net> <003001c43832$e7c25750$660918a1@fcmf.ita.br> Message-ID: <1085109623.28021.21.camel@chaos.ithaquainc.com> Rogerio, The GISToolkit does this. gistoolkit.sf.net Ithaqua. On Wed, 2004-05-12 at 10:07, Rogerio Ishibashi wrote: > Have someone a example of geotools with postigis?? > I need a example that show data (from postigis) in a map (in a java Frame) > > []'s > Rogerio Ishibashi > _______________________________________________ > postgis-devel mailing list > postgis-devel at postgis.refractions.net > http://postgis.refractions.net/mailman/listinfo/postgis-devel -- ithaqua Ithaqua Enterprises Inc.